Jul 17 21:43:47.845427 ip-10-0-131-14 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 17 21:43:47.845440 ip-10-0-131-14 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 17 21:43:47.845451 ip-10-0-131-14 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 17 21:43:47.845783 ip-10-0-131-14 systemd[1]: Failed to start Kubernetes Kubelet. Jul 17 21:43:59.453805 ip-10-0-131-14 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 17 21:43:59.453820 ip-10-0-131-14 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 93a09d4e9f45470e84642a211456b374 -- Jul 17 21:46:03.011771 ip-10-0-131-14 systemd[1]: Starting Kubernetes Kubelet... Jul 17 21:46:03.435107 ip-10-0-131-14 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 21:46:03.435107 ip-10-0-131-14 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 17 21:46:03.435107 ip-10-0-131-14 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 21:46:03.435107 ip-10-0-131-14 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 17 21:46:03.435107 ip-10-0-131-14 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 17 21:46:03.437434 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.437281 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 17 21:46:03.442526 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442509 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:46:03.442574 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442530 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:46:03.442574 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442535 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:46:03.442574 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442539 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:46:03.442574 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442544 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:46:03.442574 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442548 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:46:03.442574 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442552 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:46:03.442574 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442557 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:46:03.442574 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442561 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:46:03.442574 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442564 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:46:03.442574 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442567 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:46:03.442574 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442569 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:46:03.442574 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442572 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:46:03.442574 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442576 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:46:03.442574 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442579 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:46:03.442574 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442582 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442585 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442588 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442591 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442594 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442596 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442599 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442604 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442606 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442609 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442612 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442614 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442619 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442623 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442625 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442628 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442631 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442633 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442636 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442638 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:46:03.442964 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442641 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:46:03.443440 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442643 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:46:03.443440 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442646 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:46:03.443440 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442648 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:46:03.443440 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442651 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:46:03.443440 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442654 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:46:03.443440 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442656 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:46:03.443440 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442658 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:46:03.443440 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442661 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:46:03.443440 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442663 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:46:03.443440 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442667 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:46:03.443440 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442669 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:46:03.443440 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442672 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:46:03.443440 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442674 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:46:03.443440 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442677 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:46:03.443440 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442680 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:46:03.443440 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442682 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:46:03.443440 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442685 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:46:03.443440 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442687 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:46:03.443440 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442690 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442693 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442696 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442698 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442701 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442703 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442706 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442708 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442711 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442713 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442716 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442719 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442722 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442727 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442729 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442732 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442734 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442737 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442740 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442743 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:46:03.444036 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442745 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:46:03.444545 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442748 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:46:03.444545 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442750 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:46:03.444545 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442753 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:46:03.444545 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442756 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:46:03.444545 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442758 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:46:03.444545 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442761 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:46:03.444545 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442764 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:46:03.444545 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442767 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:46:03.444545 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442769 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:46:03.444545 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442772 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:46:03.444545 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442775 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:46:03.444545 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442777 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:46:03.444545 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442780 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:46:03.444545 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442783 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:46:03.444545 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442785 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:46:03.444545 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442788 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:46:03.444545 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442791 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:46:03.444545 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442796 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:46:03.444545 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442798 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442801 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442803 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442806 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442808 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.442811 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443563 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443570 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443573 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443576 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443579 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443582 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443585 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443588 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443591 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443593 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443596 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443599 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443601 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443604 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:46:03.445599 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443607 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:46:03.446078 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443609 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:46:03.446078 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443612 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:46:03.446078 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443615 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:46:03.446078 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443618 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:46:03.446078 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443622 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:46:03.446078 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443626 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:46:03.446078 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443629 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:46:03.446078 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443633 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:46:03.446078 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443636 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:46:03.446078 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443639 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:46:03.446078 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443642 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:46:03.446078 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443644 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:46:03.446078 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443647 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:46:03.446078 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443650 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:46:03.446078 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443652 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:46:03.446078 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443655 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:46:03.446078 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443657 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:46:03.446078 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443660 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:46:03.446078 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443663 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443667 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443671 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443674 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443677 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443679 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443682 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443686 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443688 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443691 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443694 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443696 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443699 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443702 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443704 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443707 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443709 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443712 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443715 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443717 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:46:03.446557 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443720 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:46:03.447053 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443723 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:46:03.447053 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443726 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:46:03.447053 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443728 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:46:03.447053 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443733 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:46:03.447053 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443736 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:46:03.447053 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443740 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:46:03.447053 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443742 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:46:03.447053 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443745 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:46:03.447053 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443748 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:46:03.447053 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443750 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:46:03.447053 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443753 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:46:03.447053 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443755 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:46:03.447053 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443758 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:46:03.447053 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443761 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:46:03.447053 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443763 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:46:03.447053 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443766 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:46:03.447053 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443768 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:46:03.447053 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443771 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:46:03.447053 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443773 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443775 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443778 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443780 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443783 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443785 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443789 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443791 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443793 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443796 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443798 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443809 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443812 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443815 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443817 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443820 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443823 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443825 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443828 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443831 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:46:03.447548 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443833 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443836 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443838 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443840 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443843 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443845 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.443847 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444650 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444657 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444660 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444663 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444666 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444669 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444672 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444675 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444678 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444680 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444683 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444685 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444688 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:46:03.448032 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444691 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:46:03.448540 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444693 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:46:03.448540 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444696 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:46:03.448540 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444699 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:46:03.448540 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444701 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:46:03.448540 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444704 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:46:03.448540 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444706 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:46:03.448540 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444708 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:46:03.448540 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444712 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:46:03.448540 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444716 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:46:03.448540 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444720 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:46:03.448540 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444723 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:46:03.448540 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444725 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:46:03.448540 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444728 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:46:03.448540 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444730 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:46:03.448540 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444733 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:46:03.448540 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444735 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:46:03.448540 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444738 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:46:03.448540 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444740 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:46:03.448540 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444742 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:46:03.449031 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444745 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:46:03.449031 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444747 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:46:03.449031 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444749 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:46:03.449031 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444752 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:46:03.449031 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444754 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:46:03.449031 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444756 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:46:03.449031 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444759 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:46:03.449031 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444764 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:46:03.449031 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444766 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:46:03.449031 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444769 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:46:03.449031 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444773 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:46:03.449031 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444777 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:46:03.449031 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444780 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:46:03.449031 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444783 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:46:03.449031 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444785 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:46:03.449031 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444787 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:46:03.449031 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444790 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:46:03.449031 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444792 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:46:03.449031 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444795 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:46:03.449503 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444797 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:46:03.449503 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444799 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:46:03.449503 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444802 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:46:03.449503 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444805 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:46:03.449503 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444807 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:46:03.449503 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444809 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:46:03.449503 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444812 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:46:03.449503 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444814 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:46:03.449503 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444816 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:46:03.449503 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444819 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:46:03.449503 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444822 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:46:03.449503 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444825 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:46:03.449503 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444828 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:46:03.449503 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444831 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:46:03.449503 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444834 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:46:03.449503 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444838 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:46:03.449503 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444840 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:46:03.449503 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444843 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:46:03.449503 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444845 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444847 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444851 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444853 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444856 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444859 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444861 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444864 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444866 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444869 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444871 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444874 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444876 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444878 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444881 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444883 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444885 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444888 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444891 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444893 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444895 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:46:03.449963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444898 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444900 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444902 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444905 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444907 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444910 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444912 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.444914 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.444981 2577 flags.go:64] FLAG: --address="0.0.0.0" Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.444988 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.444994 2577 flags.go:64] FLAG: --anonymous-auth="true" Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.444998 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445002 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445007 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445011 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445020 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445023 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445026 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445029 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445033 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445036 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445039 2577 flags.go:64] FLAG: --cgroup-root="" Jul 17 21:46:03.450523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445041 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445044 2577 flags.go:64] FLAG: --client-ca-file="" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445047 2577 flags.go:64] FLAG: --cloud-provider="external" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445050 2577 flags.go:64] FLAG: --cluster-dns="[]" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445054 2577 flags.go:64] FLAG: --cluster-domain="" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445057 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445060 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445063 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445066 2577 flags.go:64] FLAG: --container-log-max-files="5" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445071 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445074 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445077 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445081 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445083 2577 flags.go:64] FLAG: --contention-profiling="false" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445086 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445089 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445092 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445095 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445099 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445102 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445104 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445107 2577 flags.go:64] FLAG: --enable-load-reader="false" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445110 2577 flags.go:64] FLAG: --enable-server="true" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445114 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445118 2577 flags.go:64] FLAG: --event-burst="100" Jul 17 21:46:03.451052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445122 2577 flags.go:64] FLAG: --event-qps="50" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445125 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445128 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445131 2577 flags.go:64] FLAG: --eviction-hard="" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445135 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445139 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445142 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445144 2577 flags.go:64] FLAG: --eviction-soft="" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445147 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445150 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445153 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445156 2577 flags.go:64] FLAG: --experimental-mounter-path="" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445158 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445161 2577 flags.go:64] FLAG: --fail-swap-on="true" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445164 2577 flags.go:64] FLAG: --feature-gates="" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445168 2577 flags.go:64] FLAG: --file-check-frequency="20s" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445171 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445174 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445177 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445180 2577 flags.go:64] FLAG: --healthz-port="10248" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445182 2577 flags.go:64] FLAG: --help="false" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445185 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-131-14.ec2.internal" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445188 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445191 2577 flags.go:64] FLAG: --http-check-frequency="20s" Jul 17 21:46:03.451671 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445193 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445196 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445200 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445202 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445205 2577 flags.go:64] FLAG: --image-service-endpoint="" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445207 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445213 2577 flags.go:64] FLAG: --kube-api-burst="100" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445215 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445220 2577 flags.go:64] FLAG: --kube-api-qps="50" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445222 2577 flags.go:64] FLAG: --kube-reserved="" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445225 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445228 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445231 2577 flags.go:64] FLAG: --kubelet-cgroups="" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445234 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445237 2577 flags.go:64] FLAG: --lock-file="" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445239 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445242 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445245 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445250 2577 flags.go:64] FLAG: --log-json-split-stream="false" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445253 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445256 2577 flags.go:64] FLAG: --log-text-split-stream="false" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445258 2577 flags.go:64] FLAG: --logging-format="text" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445261 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445264 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 17 21:46:03.452284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445267 2577 flags.go:64] FLAG: --manifest-url="" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445270 2577 flags.go:64] FLAG: --manifest-url-header="" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445274 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445277 2577 flags.go:64] FLAG: --max-open-files="1000000" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445281 2577 flags.go:64] FLAG: --max-pods="110" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445283 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445286 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445289 2577 flags.go:64] FLAG: --memory-manager-policy="None" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445291 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445294 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445297 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445300 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445306 2577 flags.go:64] FLAG: --node-status-max-images="50" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445310 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445315 2577 flags.go:64] FLAG: --oom-score-adj="-999" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445318 2577 flags.go:64] FLAG: --pod-cidr="" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445322 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445327 2577 flags.go:64] FLAG: --pod-manifest-path="" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445330 2577 flags.go:64] FLAG: --pod-max-pids="-1" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445333 2577 flags.go:64] FLAG: --pods-per-core="0" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445336 2577 flags.go:64] FLAG: --port="10250" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445339 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445342 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-032880f3b07e50aed" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445345 2577 flags.go:64] FLAG: --qos-reserved="" Jul 17 21:46:03.452886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445348 2577 flags.go:64] FLAG: --read-only-port="10255" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445350 2577 flags.go:64] FLAG: --register-node="true" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445353 2577 flags.go:64] FLAG: --register-with-taints="" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445357 2577 flags.go:64] FLAG: --registry-burst="10" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445360 2577 flags.go:64] FLAG: --registry-qps="5" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445363 2577 flags.go:64] FLAG: --reserved-cpus="" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445366 2577 flags.go:64] FLAG: --reserved-memory="" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445369 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445372 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445376 2577 flags.go:64] FLAG: --rotate-certificates="false" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445379 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445381 2577 flags.go:64] FLAG: --runonce="false" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445384 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445387 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445389 2577 flags.go:64] FLAG: --seccomp-default="false" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445392 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445395 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445398 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445400 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445403 2577 flags.go:64] FLAG: --storage-driver-password="root" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445406 2577 flags.go:64] FLAG: --storage-driver-secure="false" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445409 2577 flags.go:64] FLAG: --storage-driver-table="stats" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445413 2577 flags.go:64] FLAG: --storage-driver-user="root" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445416 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445419 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 17 21:46:03.453462 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445423 2577 flags.go:64] FLAG: --system-cgroups="" Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445425 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445431 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445434 2577 flags.go:64] FLAG: --tls-cert-file="" Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445436 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445440 2577 flags.go:64] FLAG: --tls-min-version="" Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445442 2577 flags.go:64] FLAG: --tls-private-key-file="" Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445445 2577 flags.go:64] FLAG: --topology-manager-policy="none" Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445448 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445451 2577 flags.go:64] FLAG: --topology-manager-scope="container" Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445454 2577 flags.go:64] FLAG: --v="2" Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445457 2577 flags.go:64] FLAG: --version="false" Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445461 2577 flags.go:64] FLAG: --vmodule="" Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445465 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.445468 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445587 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445591 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445594 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445597 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445600 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445602 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445605 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445607 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:46:03.454297 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445610 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:46:03.455254 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445613 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:46:03.455254 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445615 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:46:03.455254 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445617 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:46:03.455254 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445620 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:46:03.455254 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445622 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:46:03.455254 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445625 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:46:03.455254 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445628 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:46:03.455254 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445630 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:46:03.455254 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445634 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:46:03.455254 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445636 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:46:03.455254 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445639 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:46:03.455254 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445641 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:46:03.455254 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445644 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:46:03.455254 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445647 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:46:03.455254 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445649 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:46:03.455254 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445652 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:46:03.455254 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445654 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:46:03.455254 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445656 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:46:03.455254 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445659 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445661 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445663 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445666 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445668 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445670 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445673 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445675 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445678 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445680 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445683 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445685 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445688 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445690 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445692 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445694 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445697 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445699 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445702 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445704 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:46:03.455754 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445707 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445709 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445713 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445715 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445717 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445721 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445725 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445728 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445731 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445733 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445735 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445738 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445740 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445742 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445745 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445747 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445750 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445752 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445755 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445757 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:46:03.456252 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445760 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:46:03.456799 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445763 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:46:03.456799 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445765 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:46:03.456799 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445767 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:46:03.456799 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445770 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:46:03.456799 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445772 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:46:03.456799 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445774 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:46:03.456799 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445778 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:46:03.456799 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445782 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:46:03.456799 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445784 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:46:03.456799 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445787 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:46:03.456799 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445791 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:46:03.456799 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445794 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:46:03.456799 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445797 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:46:03.456799 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445802 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:46:03.456799 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445804 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:46:03.456799 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445807 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:46:03.456799 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445809 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:46:03.456799 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445812 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:46:03.456799 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445815 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:46:03.457257 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445817 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:46:03.457257 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445820 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:46:03.457257 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445822 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:46:03.457257 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445825 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:46:03.457257 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445827 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:46:03.457257 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445830 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:46:03.457257 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445832 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:46:03.457257 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445835 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:46:03.457257 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445837 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:46:03.457257 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445840 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:46:03.457257 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445842 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:46:03.457257 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.445845 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:46:03.457257 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.446398 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 21:46:03.457257 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.456470 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 17 21:46:03.457257 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.456502 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 17 21:46:03.457649 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.456522 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 17 21:46:03.457649 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.456528 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 17 21:46:03.457649 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456601 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:46:03.457649 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456607 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:46:03.457649 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456611 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:46:03.457649 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456614 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:46:03.457649 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456617 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:46:03.457649 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456619 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:46:03.457649 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456622 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:46:03.457649 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456624 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:46:03.457649 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456627 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:46:03.457649 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456630 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:46:03.457649 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456633 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:46:03.457649 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456635 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:46:03.457649 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456638 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:46:03.457649 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456640 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:46:03.457649 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456643 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:46:03.457649 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456645 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:46:03.457649 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456648 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456650 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456652 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456655 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456657 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456660 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456662 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456665 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456667 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456670 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456673 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456676 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456678 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456681 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456685 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456689 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456692 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456695 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456698 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456701 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:46:03.458108 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456703 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456706 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456708 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456711 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456713 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456716 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456718 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456720 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456723 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456725 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456728 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456730 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456732 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456736 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456739 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456742 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456745 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456748 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456751 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456753 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:46:03.458696 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456756 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456759 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456762 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456765 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456767 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456770 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456773 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456776 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456778 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456780 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456783 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456785 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456788 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456790 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456792 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456795 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456797 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456799 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456802 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456804 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:46:03.459172 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456807 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456810 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456812 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456815 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456818 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456820 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456822 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456825 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456828 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456830 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456832 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456835 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456837 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456839 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456842 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456846 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456850 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456853 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456856 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456858 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:46:03.459662 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456860 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:46:03.460155 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456863 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:46:03.460155 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456866 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:46:03.460155 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.456871 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 21:46:03.460155 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456991 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 17 21:46:03.460155 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456996 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 17 21:46:03.460155 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.456999 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 17 21:46:03.460155 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457002 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 17 21:46:03.460155 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457005 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 17 21:46:03.460155 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457008 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 17 21:46:03.460155 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457011 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 17 21:46:03.460155 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457014 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 17 21:46:03.460155 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457019 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 17 21:46:03.460155 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457023 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 17 21:46:03.460155 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457025 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457028 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457031 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457035 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457038 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457040 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457043 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457046 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457049 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457051 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457054 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457057 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457060 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457062 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457065 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457068 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457070 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457073 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457075 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457077 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 17 21:46:03.460512 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457080 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 17 21:46:03.460991 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457082 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 17 21:46:03.460991 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457085 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 17 21:46:03.460991 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457087 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 17 21:46:03.460991 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457089 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 17 21:46:03.460991 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457092 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 17 21:46:03.460991 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457094 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 17 21:46:03.460991 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457096 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 17 21:46:03.460991 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457098 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 17 21:46:03.460991 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457101 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 17 21:46:03.460991 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457104 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 17 21:46:03.460991 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457106 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 17 21:46:03.460991 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457108 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 17 21:46:03.460991 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457110 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 17 21:46:03.460991 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457113 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 17 21:46:03.460991 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457116 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 17 21:46:03.460991 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457118 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 17 21:46:03.460991 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457120 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 17 21:46:03.460991 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457123 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 17 21:46:03.460991 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457125 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457128 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457130 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457132 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457135 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457137 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457140 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457142 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457145 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457147 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457150 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457152 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457155 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457157 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457160 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457162 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457165 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457167 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457170 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457172 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 17 21:46:03.461442 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457174 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457177 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457179 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457181 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457184 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457186 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457189 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457191 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457193 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457196 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457199 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457201 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457203 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457206 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457208 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457211 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457215 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457217 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457220 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457223 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 17 21:46:03.461936 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457226 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 17 21:46:03.462408 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457228 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 17 21:46:03.462408 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457231 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 17 21:46:03.462408 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457233 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 17 21:46:03.462408 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457235 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 17 21:46:03.462408 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457238 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 17 21:46:03.462408 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457240 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 17 21:46:03.462408 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457243 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 17 21:46:03.462408 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457245 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 17 21:46:03.462408 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:03.457247 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 17 21:46:03.462408 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.457253 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 17 21:46:03.462408 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.457366 2577 server.go:962] "Client rotation is on, will bootstrap in background" Jul 17 21:46:03.462408 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.460170 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 17 21:46:03.462408 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.461600 2577 server.go:1019] "Starting client certificate rotation" Jul 17 21:46:03.462408 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.461698 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 21:46:03.462822 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.461735 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 17 21:46:03.463957 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.463944 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 21:46:03.465978 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.465963 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 17 21:46:03.475420 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.475403 2577 log.go:25] "Validated CRI v1 runtime API" Jul 17 21:46:03.479925 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.479911 2577 log.go:25] "Validated CRI v1 image API" Jul 17 21:46:03.481415 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.481398 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 17 21:46:03.484676 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.484645 2577 fs.go:135] Filesystem UUIDs: map[2136aacf-73e1-4247-b989-8222e13afa22:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 8ece64c9-995e-4f1b-9aab-674cecb84b50:/dev/nvme0n1p3] Jul 17 21:46:03.484754 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.484675 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 17 21:46:03.489603 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.489478 2577 manager.go:216] Machine: {Timestamp:2026-07-17 21:46:03.488174298 +0000 UTC m=+0.348397811 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100959 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2789e11c458302979daa92a548842c SystemUUID:ec2789e1-1c45-8302-979d-aa92a548842c BootID:93a09d4e-9f45-470e-8464-2a211456b374 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:05:be:ae:64:91 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:05:be:ae:64:91 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:69:a2:fc:a9:0a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 17 21:46:03.489603 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.489593 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 17 21:46:03.489730 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.489669 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 17 21:46:03.491193 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.491177 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 21:46:03.493474 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.493191 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 17 21:46:03.493640 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.493479 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-14.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 17 21:46:03.494210 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.494199 2577 topology_manager.go:138] "Creating topology manager with none policy" Jul 17 21:46:03.494246 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.494213 2577 container_manager_linux.go:309] "Creating device plugin manager" Jul 17 21:46:03.494246 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.494221 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 21:46:03.494881 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.494870 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 17 21:46:03.496533 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.496513 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 17 21:46:03.499781 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.499770 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 17 21:46:03.499907 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.499897 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 17 21:46:03.499939 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.499927 2577 kubelet.go:486] "Attempting to sync node with API server" Jul 17 21:46:03.499939 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.499937 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 17 21:46:03.499996 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.499948 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 17 21:46:03.499996 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.499958 2577 kubelet.go:398] "Adding apiserver pod source" Jul 17 21:46:03.499996 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.499966 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 17 21:46:03.500363 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.500352 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 21:46:03.500409 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.500370 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 17 21:46:03.502722 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.502708 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 17 21:46:03.504583 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.504570 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 17 21:46:03.504630 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.504601 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 17 21:46:03.505733 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.505718 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 17 21:46:03.505799 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.505739 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 17 21:46:03.505799 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.505748 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 17 21:46:03.505799 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.505756 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 17 21:46:03.505799 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.505768 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 17 21:46:03.505799 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.505776 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 17 21:46:03.505799 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.505788 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 17 21:46:03.505799 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.505796 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 17 21:46:03.506018 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.505806 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 17 21:46:03.506018 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.505814 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 17 21:46:03.506018 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.505827 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 17 21:46:03.506018 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.505836 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 17 21:46:03.506648 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.506635 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 17 21:46:03.506700 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.506650 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 17 21:46:03.509869 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.509853 2577 server.go:1268] "Started kubelet" Jul 17 21:46:03.510010 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.509939 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 17 21:46:03.510070 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.510029 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 17 21:46:03.510070 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.509981 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 17 21:46:03.510589 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.510568 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-14.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 17 21:46:03.510668 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:03.510593 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-14.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 17 21:46:03.510668 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:03.510647 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 17 21:46:03.510757 ip-10-0-131-14 systemd[1]: Started Kubernetes Kubelet. Jul 17 21:46:03.512000 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.511986 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 17 21:46:03.512099 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.512084 2577 server.go:310] "Adding debug handlers to kubelet server" Jul 17 21:46:03.515433 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:03.515403 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 17 21:46:03.515857 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.515843 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d4kn2" Jul 17 21:46:03.517082 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:03.516278 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-14.ec2.internal.18c331eb2cefa307 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-14.ec2.internal,UID:ip-10-0-131-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-14.ec2.internal,},FirstTimestamp:2026-07-17 21:46:03.509826311 +0000 UTC m=+0.370049804,LastTimestamp:2026-07-17 21:46:03.509826311 +0000 UTC m=+0.370049804,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-14.ec2.internal,}" Jul 17 21:46:03.518421 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.518402 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 17 21:46:03.518421 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.518408 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 17 21:46:03.518659 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:03.518641 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Jul 17 21:46:03.519080 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.519067 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 17 21:46:03.519362 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.519345 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 17 21:46:03.519362 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.519362 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 17 21:46:03.519520 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.519469 2577 reconstruct.go:97] "Volume reconstruction finished" Jul 17 21:46:03.519520 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.519480 2577 reconciler.go:29] "Reconciler: start to sync state" Jul 17 21:46:03.521090 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.521068 2577 factory.go:55] Registering systemd factory Jul 17 21:46:03.521090 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.521093 2577 factory.go:223] Registration of the systemd container factory successfully Jul 17 21:46:03.521471 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.521440 2577 factory.go:153] Registering CRI-O factory Jul 17 21:46:03.521471 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.521460 2577 factory.go:223] Registration of the crio container factory successfully Jul 17 21:46:03.521603 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.521540 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 17 21:46:03.521657 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.521606 2577 factory.go:103] Registering Raw factory Jul 17 21:46:03.521657 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.521623 2577 manager.go:1196] Started watching for new ooms in manager Jul 17 21:46:03.522102 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:03.522066 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-14.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 17 21:46:03.522172 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:03.522145 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 17 21:46:03.522426 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.522414 2577 manager.go:349] Starting recovery of all containers Jul 17 21:46:03.523035 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.523017 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-d4kn2" Jul 17 21:46:03.535092 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.534979 2577 manager.go:354] Recovery completed Jul 17 21:46:03.539132 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.539117 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:46:03.541004 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.540987 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:46:03.541079 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.541021 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:46:03.541079 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.541036 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:46:03.541640 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.541624 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 17 21:46:03.541640 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.541633 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:46:03.541640 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.541639 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 17 21:46:03.541748 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.541657 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 17 21:46:03.542309 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.542292 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:46:03.542379 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.542326 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:46:03.542379 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.542339 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:46:03.542768 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.542755 2577 policy_none.go:49] "None policy: Start" Jul 17 21:46:03.542848 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.542770 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 17 21:46:03.542848 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.542783 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 17 21:46:03.542986 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:03.542904 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-14.ec2.internal.18c331eb2ecb6019 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-14.ec2.internal,UID:ip-10-0-131-14.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-14.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-14.ec2.internal,},FirstTimestamp:2026-07-17 21:46:03.541004313 +0000 UTC m=+0.401227814,LastTimestamp:2026-07-17 21:46:03.541004313 +0000 UTC m=+0.401227814,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-14.ec2.internal,}" Jul 17 21:46:03.543877 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.543865 2577 policy_none.go:47] "Start" Jul 17 21:46:03.576035 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.576017 2577 manager.go:337] "Starting Device Plugin manager" Jul 17 21:46:03.576124 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:03.576049 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 17 21:46:03.576124 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.576058 2577 server.go:87] "Starting device plugin registration server" Jul 17 21:46:03.576303 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.576290 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 17 21:46:03.576365 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.576302 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 17 21:46:03.576418 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.576399 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 17 21:46:03.576528 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.576470 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 17 21:46:03.576528 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.576477 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 17 21:46:03.577004 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:03.576986 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 17 21:46:03.577073 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:03.577054 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-14.ec2.internal\" not found" Jul 17 21:46:03.676968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.676924 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:46:03.678272 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.678139 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:46:03.678272 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.678174 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:46:03.678272 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.678186 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:46:03.678272 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.678215 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-14.ec2.internal" Jul 17 21:46:03.686191 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.686135 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-14.ec2.internal" Jul 17 21:46:03.686191 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:03.686163 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-14.ec2.internal\": node \"ip-10-0-131-14.ec2.internal\" not found" Jul 17 21:46:03.692123 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.692094 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 17 21:46:03.693481 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.693460 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 17 21:46:03.693614 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.693524 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 17 21:46:03.693614 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.693552 2577 kubelet.go:2448] "Starting kubelet main sync loop" Jul 17 21:46:03.693614 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:03.693586 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 17 21:46:03.695690 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.695669 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:46:03.702427 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:03.702407 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Jul 17 21:46:03.794327 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.794291 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal"] Jul 17 21:46:03.794441 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.794371 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:46:03.795393 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.795376 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:46:03.795465 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.795408 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:46:03.795465 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.795423 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:46:03.796798 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.796782 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:46:03.796898 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.796884 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" Jul 17 21:46:03.796943 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.796922 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:46:03.797526 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.797511 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:46:03.797599 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.797546 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:46:03.797599 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.797556 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:46:03.797599 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.797582 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:46:03.797693 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.797602 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:46:03.797693 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.797617 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:46:03.798890 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.798876 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" Jul 17 21:46:03.798938 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.798909 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 17 21:46:03.799663 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.799646 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientMemory" Jul 17 21:46:03.799775 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.799680 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasNoDiskPressure" Jul 17 21:46:03.799775 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.799693 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeHasSufficientPID" Jul 17 21:46:03.802578 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:03.802563 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Jul 17 21:46:03.821094 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.821071 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30e4b77ac143ec43bfae8d55663eb347-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal\" (UID: \"30e4b77ac143ec43bfae8d55663eb347\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" Jul 17 21:46:03.821223 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.821100 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30e4b77ac143ec43bfae8d55663eb347-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal\" (UID: \"30e4b77ac143ec43bfae8d55663eb347\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" Jul 17 21:46:03.821223 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.821117 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09f0258f09213a85d398e0f2eb07ae5f-config\") pod \"kube-apiserver-proxy-ip-10-0-131-14.ec2.internal\" (UID: \"09f0258f09213a85d398e0f2eb07ae5f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" Jul 17 21:46:03.821223 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:03.821200 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-14.ec2.internal\" not found" node="ip-10-0-131-14.ec2.internal" Jul 17 21:46:03.825616 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:03.825601 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-14.ec2.internal\" not found" node="ip-10-0-131-14.ec2.internal" Jul 17 21:46:03.902878 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:03.902843 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Jul 17 21:46:03.921466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.921438 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30e4b77ac143ec43bfae8d55663eb347-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal\" (UID: \"30e4b77ac143ec43bfae8d55663eb347\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" Jul 17 21:46:03.921653 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.921474 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09f0258f09213a85d398e0f2eb07ae5f-config\") pod \"kube-apiserver-proxy-ip-10-0-131-14.ec2.internal\" (UID: \"09f0258f09213a85d398e0f2eb07ae5f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" Jul 17 21:46:03.921653 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.921531 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/09f0258f09213a85d398e0f2eb07ae5f-config\") pod \"kube-apiserver-proxy-ip-10-0-131-14.ec2.internal\" (UID: \"09f0258f09213a85d398e0f2eb07ae5f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" Jul 17 21:46:03.921653 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.921531 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/30e4b77ac143ec43bfae8d55663eb347-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal\" (UID: \"30e4b77ac143ec43bfae8d55663eb347\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" Jul 17 21:46:03.921653 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.921564 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30e4b77ac143ec43bfae8d55663eb347-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal\" (UID: \"30e4b77ac143ec43bfae8d55663eb347\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" Jul 17 21:46:03.921653 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:03.921584 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/30e4b77ac143ec43bfae8d55663eb347-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal\" (UID: \"30e4b77ac143ec43bfae8d55663eb347\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" Jul 17 21:46:04.003323 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:04.003244 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Jul 17 21:46:04.103783 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:04.103736 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Jul 17 21:46:04.124211 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:04.124190 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" Jul 17 21:46:04.129178 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:04.129161 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" Jul 17 21:46:04.204144 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:04.204105 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Jul 17 21:46:04.304682 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:04.304609 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Jul 17 21:46:04.405146 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:04.405110 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Jul 17 21:46:04.460630 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:04.460604 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 17 21:46:04.461105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:04.460751 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 17 21:46:04.506041 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:04.506007 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Jul 17 21:46:04.530867 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:04.530837 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 21:41:03 +0000 UTC" deadline="2028-02-13 21:45:05.399427995 +0000 UTC" Jul 17 21:46:04.530867 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:04.530862 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13823h59m0.868569095s" Jul 17 21:46:04.574929 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:04.574864 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:46:04.606160 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:04.606138 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Jul 17 21:46:04.614574 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:04.614553 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:46:04.707139 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:04.707111 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Jul 17 21:46:04.732818 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:04.732786 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f0258f09213a85d398e0f2eb07ae5f.slice/crio-764a84f37a426c58f82bba91e1dc115aced7622b6923e4b7f28739bc46ce48ab WatchSource:0}: Error finding container 764a84f37a426c58f82bba91e1dc115aced7622b6923e4b7f28739bc46ce48ab: Status 404 returned error can't find the container with id 764a84f37a426c58f82bba91e1dc115aced7622b6923e4b7f28739bc46ce48ab Jul 17 21:46:04.733380 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:04.733354 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e4b77ac143ec43bfae8d55663eb347.slice/crio-20a37666ea33792aa309c7808013fb0dcfd786b9d880bed850da8611420e5ac0 WatchSource:0}: Error finding container 20a37666ea33792aa309c7808013fb0dcfd786b9d880bed850da8611420e5ac0: Status 404 returned error can't find the container with id 20a37666ea33792aa309c7808013fb0dcfd786b9d880bed850da8611420e5ac0 Jul 17 21:46:04.739466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:04.739449 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 21:46:04.807824 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:04.807789 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-14.ec2.internal\" not found" Jul 17 21:46:04.856313 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:04.856261 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:46:04.919666 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:04.919643 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" Jul 17 21:46:04.936105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:04.936088 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 21:46:04.937232 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:04.937218 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" Jul 17 21:46:04.946571 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:04.946555 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 17 21:46:05.500976 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.500936 2577 apiserver.go:52] "Watching apiserver" Jul 17 21:46:05.507993 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.507960 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 17 21:46:05.508399 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.508372 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-j6dqx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal","openshift-multus/multus-additional-cni-plugins-fcfh7","openshift-multus/multus-kdmxg","openshift-multus/network-metrics-daemon-4vfnr","openshift-network-diagnostics/network-check-target-kp5bb","kube-system/konnectivity-agent-97mzp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n","openshift-cluster-node-tuning-operator/tuned-h5xhb","openshift-image-registry/node-ca-26z9v","openshift-network-operator/iptables-alerter-wxxlz","openshift-ovn-kubernetes/ovnkube-node-46wvc","kube-system/global-pull-secret-syncer-982gr","kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal"] Jul 17 21:46:05.510105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.510081 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.512181 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.512156 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:05.512289 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:05.512261 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:05.513164 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.513141 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 17 21:46:05.513729 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.513705 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 17 21:46:05.513873 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.513739 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-97mzp" Jul 17 21:46:05.514312 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.514291 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 17 21:46:05.514426 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.514323 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cjbwk\"" Jul 17 21:46:05.514426 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.514324 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 17 21:46:05.515321 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.515303 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.516721 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.516703 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:05.516833 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:05.516804 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:05.518861 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.518410 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 17 21:46:05.518861 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.518422 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p6hkn\"" Jul 17 21:46:05.518861 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.518425 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 17 21:46:05.518861 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.518638 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 17 21:46:05.518861 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.518706 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 17 21:46:05.518861 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.518746 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 17 21:46:05.519225 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.518988 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fd6sd\"" Jul 17 21:46:05.519225 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.519208 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j6dqx" Jul 17 21:46:05.520487 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.520178 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.521392 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.521330 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 17 21:46:05.521583 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.521490 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 17 21:46:05.521691 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.521674 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.521796 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.521779 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sm6md\"" Jul 17 21:46:05.522628 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.522607 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 17 21:46:05.522927 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.522885 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 17 21:46:05.522927 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.522909 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-769p8\"" Jul 17 21:46:05.523168 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.523149 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 17 21:46:05.523300 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.523248 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-26z9v" Jul 17 21:46:05.524125 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.524103 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pxs2l\"" Jul 17 21:46:05.524590 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.524343 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 17 21:46:05.525637 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.524943 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 17 21:46:05.525637 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.525013 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wxxlz" Jul 17 21:46:05.526265 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.526027 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 17 21:46:05.526265 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.526114 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 17 21:46:05.526820 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.526656 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vcsfw\"" Jul 17 21:46:05.527016 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.526888 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 17 21:46:05.529477 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.529073 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.529477 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.529186 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 17 21:46:05.529477 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.529260 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sc5s8\"" Jul 17 21:46:05.529700 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.529494 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 17 21:46:05.529900 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.529881 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 17 21:46:05.530749 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.530731 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:05.530857 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:05.530801 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:05.532574 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.532548 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 17 21:46:05.532670 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.532594 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 17 21:46:05.532870 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.532846 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-host-run-netns\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.532979 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.532885 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/171b015d-bd82-4094-b923-27cddda2aa06-etc-selinux\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.532979 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.532900 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 17 21:46:05.532979 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.532915 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/55dc68af-baa5-4429-a748-6a5ed468c1b3-hosts-file\") pod \"node-resolver-j6dqx\" (UID: \"55dc68af-baa5-4429-a748-6a5ed468c1b3\") " pod="openshift-dns/node-resolver-j6dqx" Jul 17 21:46:05.532979 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.532940 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55dc68af-baa5-4429-a748-6a5ed468c1b3-tmp-dir\") pod \"node-resolver-j6dqx\" (UID: \"55dc68af-baa5-4429-a748-6a5ed468c1b3\") " pod="openshift-dns/node-resolver-j6dqx" Jul 17 21:46:05.532979 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.532968 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f07efcef-f35f-45b5-873c-28d5acedb9e0-agent-certs\") pod \"konnectivity-agent-97mzp\" (UID: \"f07efcef-f35f-45b5-873c-28d5acedb9e0\") " pod="kube-system/konnectivity-agent-97mzp" Jul 17 21:46:05.533211 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533008 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1ac35ab-4786-4165-8b80-33cd8f78a74b-os-release\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.533211 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533036 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1ac35ab-4786-4165-8b80-33cd8f78a74b-cni-binary-copy\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.533211 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533062 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1ac35ab-4786-4165-8b80-33cd8f78a74b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.533211 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533127 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 17 21:46:05.533211 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533131 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-multus-cni-dir\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.533211 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533127 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 17 21:46:05.533211 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533185 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 17 21:46:05.533577 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533185 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-host-var-lib-kubelet\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.533577 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533262 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-host-run-multus-certs\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.533577 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533286 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs\") pod \"network-metrics-daemon-4vfnr\" (UID: \"491a8723-75cd-4309-abad-792b39758916\") " pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:05.533577 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533312 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1ac35ab-4786-4165-8b80-33cd8f78a74b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.533577 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533339 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/171b015d-bd82-4094-b923-27cddda2aa06-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.533577 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533363 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/171b015d-bd82-4094-b923-27cddda2aa06-sys-fs\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.533577 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533391 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-etc-kubernetes\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.533577 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533442 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1ac35ab-4786-4165-8b80-33cd8f78a74b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.533577 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533529 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9744\" (UniqueName: \"kubernetes.io/projected/171b015d-bd82-4094-b923-27cddda2aa06-kube-api-access-c9744\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.533577 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533560 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-cnibin\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.534100 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533589 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-host-var-lib-cni-multus\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.534100 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533615 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-hostroot\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.534100 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533639 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sbtp\" (UniqueName: \"kubernetes.io/projected/cb155f63-69cf-4b8c-ad77-324207fefb52-kube-api-access-8sbtp\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.534100 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533665 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxj2n\" (UniqueName: \"kubernetes.io/projected/491a8723-75cd-4309-abad-792b39758916-kube-api-access-sxj2n\") pod \"network-metrics-daemon-4vfnr\" (UID: \"491a8723-75cd-4309-abad-792b39758916\") " pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:05.534100 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533688 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1ac35ab-4786-4165-8b80-33cd8f78a74b-cnibin\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.534100 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533711 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/171b015d-bd82-4094-b923-27cddda2aa06-device-dir\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.534100 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-host-run-k8s-cni-cncf-io\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.534100 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533769 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-os-release\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.534100 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533811 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9brjf\"" Jul 17 21:46:05.534100 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533831 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb155f63-69cf-4b8c-ad77-324207fefb52-cni-binary-copy\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.534100 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533873 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-multus-socket-dir-parent\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.534100 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533910 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-host-var-lib-cni-bin\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.534100 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533952 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1ac35ab-4786-4165-8b80-33cd8f78a74b-system-cni-dir\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.534100 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.533977 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cpmq\" (UniqueName: \"kubernetes.io/projected/e1ac35ab-4786-4165-8b80-33cd8f78a74b-kube-api-access-6cpmq\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.534100 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.534005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/171b015d-bd82-4094-b923-27cddda2aa06-socket-dir\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.534100 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.534025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb155f63-69cf-4b8c-ad77-324207fefb52-multus-daemon-config\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.534100 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.534041 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f07efcef-f35f-45b5-873c-28d5acedb9e0-konnectivity-ca\") pod \"konnectivity-agent-97mzp\" (UID: \"f07efcef-f35f-45b5-873c-28d5acedb9e0\") " pod="kube-system/konnectivity-agent-97mzp" Jul 17 21:46:05.534835 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.534057 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/171b015d-bd82-4094-b923-27cddda2aa06-registration-dir\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.534835 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.534073 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qngkj\" (UniqueName: \"kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj\") pod \"network-check-target-kp5bb\" (UID: \"ee877ecd-672d-4415-a69e-831da8e098fe\") " pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:05.534835 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.534088 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xsg9\" (UniqueName: \"kubernetes.io/projected/55dc68af-baa5-4429-a748-6a5ed468c1b3-kube-api-access-8xsg9\") pod \"node-resolver-j6dqx\" (UID: \"55dc68af-baa5-4429-a748-6a5ed468c1b3\") " pod="openshift-dns/node-resolver-j6dqx" Jul 17 21:46:05.534835 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.534109 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-system-cni-dir\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.534835 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.534128 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-multus-conf-dir\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.538842 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.538823 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 17 21:46:05.570717 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.570697 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4t6zb" Jul 17 21:46:05.577749 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.577730 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4t6zb" Jul 17 21:46:05.620429 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.620406 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 17 21:46:05.635357 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635325 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-system-cni-dir\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.635601 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635363 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-multus-conf-dir\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.635601 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635388 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-modprobe-d\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.635601 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635453 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-multus-conf-dir\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.635601 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635458 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-run\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.635601 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635471 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-system-cni-dir\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.635601 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635543 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f038fc3f-ef28-4567-bc5a-0d583093a80e-iptables-alerter-script\") pod \"iptables-alerter-wxxlz\" (UID: \"f038fc3f-ef28-4567-bc5a-0d583093a80e\") " pod="openshift-network-operator/iptables-alerter-wxxlz" Jul 17 21:46:05.635601 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635602 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/171b015d-bd82-4094-b923-27cddda2aa06-etc-selinux\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.636160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635630 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-var-lib-kubelet\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.636160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635656 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-host\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.636160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635681 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgd69\" (UniqueName: \"kubernetes.io/projected/f06842a6-0e34-43ba-a98a-ffc1f6501294-kube-api-access-mgd69\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.636160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635709 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1ac35ab-4786-4165-8b80-33cd8f78a74b-cni-binary-copy\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.636160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635735 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wwpc\" (UniqueName: \"kubernetes.io/projected/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-kube-api-access-2wwpc\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.636160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635762 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret\") pod \"global-pull-secret-syncer-982gr\" (UID: \"e2430562-9210-4afd-9800-9e66f9548cb7\") " pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:05.636160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635790 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/171b015d-bd82-4094-b923-27cddda2aa06-etc-selinux\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.636160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-kubernetes\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.636160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635839 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-tuned\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.636160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635866 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f06842a6-0e34-43ba-a98a-ffc1f6501294-tmp\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.636160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635891 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-run-ovn-kubernetes\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.636160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635921 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-multus-cni-dir\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.636160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.635969 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-host-run-multus-certs\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.636160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs\") pod \"network-metrics-daemon-4vfnr\" (UID: \"491a8723-75cd-4309-abad-792b39758916\") " pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:05.636160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636063 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-run-systemd\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.636160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636109 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-multus-cni-dir\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.636160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636097 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-etc-openvswitch\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.637466 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:05.636141 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:46:05.637466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636166 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-etc-kubernetes\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.637466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636170 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-host-run-multus-certs\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.637466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636195 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1ac35ab-4786-4165-8b80-33cd8f78a74b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.637466 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:05.636238 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs podName:491a8723-75cd-4309-abad-792b39758916 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:06.136196181 +0000 UTC m=+2.996419679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs") pod "network-metrics-daemon-4vfnr" (UID: "491a8723-75cd-4309-abad-792b39758916") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:46:05.637466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636233 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-etc-kubernetes\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.637466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9744\" (UniqueName: \"kubernetes.io/projected/171b015d-bd82-4094-b923-27cddda2aa06-kube-api-access-c9744\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.637466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636429 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1ac35ab-4786-4165-8b80-33cd8f78a74b-cni-binary-copy\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.637466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636430 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-host-var-lib-cni-multus\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.637466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636473 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-host-var-lib-cni-multus\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.637466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636519 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sbtp\" (UniqueName: \"kubernetes.io/projected/cb155f63-69cf-4b8c-ad77-324207fefb52-kube-api-access-8sbtp\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.637466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxj2n\" (UniqueName: \"kubernetes.io/projected/491a8723-75cd-4309-abad-792b39758916-kube-api-access-sxj2n\") pod \"network-metrics-daemon-4vfnr\" (UID: \"491a8723-75cd-4309-abad-792b39758916\") " pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:05.637466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636613 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-cni-bin\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.637466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636660 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-ovn-node-metrics-cert\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.637466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636700 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2430562-9210-4afd-9800-9e66f9548cb7-dbus\") pod \"global-pull-secret-syncer-982gr\" (UID: \"e2430562-9210-4afd-9800-9e66f9548cb7\") " pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:05.637466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636730 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb155f63-69cf-4b8c-ad77-324207fefb52-cni-binary-copy\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.637466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636852 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-multus-socket-dir-parent\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.638383 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636865 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1ac35ab-4786-4165-8b80-33cd8f78a74b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.638383 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636894 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1ac35ab-4786-4165-8b80-33cd8f78a74b-system-cni-dir\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.638383 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636939 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/171b015d-bd82-4094-b923-27cddda2aa06-socket-dir\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.638383 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636980 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-run-ovn\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.638383 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637003 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-multus-socket-dir-parent\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.638383 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.636945 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1ac35ab-4786-4165-8b80-33cd8f78a74b-system-cni-dir\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.638383 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637007 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-ovnkube-config\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.638383 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637057 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2430562-9210-4afd-9800-9e66f9548cb7-kubelet-config\") pod \"global-pull-secret-syncer-982gr\" (UID: \"e2430562-9210-4afd-9800-9e66f9548cb7\") " pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:05.638383 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637086 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/171b015d-bd82-4094-b923-27cddda2aa06-socket-dir\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.638383 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637094 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb155f63-69cf-4b8c-ad77-324207fefb52-multus-daemon-config\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.638383 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f07efcef-f35f-45b5-873c-28d5acedb9e0-konnectivity-ca\") pod \"konnectivity-agent-97mzp\" (UID: \"f07efcef-f35f-45b5-873c-28d5acedb9e0\") " pod="kube-system/konnectivity-agent-97mzp" Jul 17 21:46:05.638383 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qngkj\" (UniqueName: \"kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj\") pod \"network-check-target-kp5bb\" (UID: \"ee877ecd-672d-4415-a69e-831da8e098fe\") " pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:05.638383 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637213 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-kubelet\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.638383 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xsg9\" (UniqueName: \"kubernetes.io/projected/55dc68af-baa5-4429-a748-6a5ed468c1b3-kube-api-access-8xsg9\") pod \"node-resolver-j6dqx\" (UID: \"55dc68af-baa5-4429-a748-6a5ed468c1b3\") " pod="openshift-dns/node-resolver-j6dqx" Jul 17 21:46:05.638383 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637323 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-node-log\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.638383 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637350 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxkgh\" (UniqueName: \"kubernetes.io/projected/f038fc3f-ef28-4567-bc5a-0d583093a80e-kube-api-access-dxkgh\") pod \"iptables-alerter-wxxlz\" (UID: \"f038fc3f-ef28-4567-bc5a-0d583093a80e\") " pod="openshift-network-operator/iptables-alerter-wxxlz" Jul 17 21:46:05.638383 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637379 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-host-run-netns\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.639291 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637409 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-run-openvswitch\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.639291 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637434 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cb155f63-69cf-4b8c-ad77-324207fefb52-cni-binary-copy\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.639291 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637461 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-host-run-netns\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.639291 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637487 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cb155f63-69cf-4b8c-ad77-324207fefb52-multus-daemon-config\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.639291 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637491 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-cni-netd\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.639291 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637540 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-sysctl-conf\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.639291 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/55dc68af-baa5-4429-a748-6a5ed468c1b3-hosts-file\") pod \"node-resolver-j6dqx\" (UID: \"55dc68af-baa5-4429-a748-6a5ed468c1b3\") " pod="openshift-dns/node-resolver-j6dqx" Jul 17 21:46:05.639291 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637612 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55dc68af-baa5-4429-a748-6a5ed468c1b3-tmp-dir\") pod \"node-resolver-j6dqx\" (UID: \"55dc68af-baa5-4429-a748-6a5ed468c1b3\") " pod="openshift-dns/node-resolver-j6dqx" Jul 17 21:46:05.639291 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f07efcef-f35f-45b5-873c-28d5acedb9e0-agent-certs\") pod \"konnectivity-agent-97mzp\" (UID: \"f07efcef-f35f-45b5-873c-28d5acedb9e0\") " pod="kube-system/konnectivity-agent-97mzp" Jul 17 21:46:05.639291 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637665 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1ac35ab-4786-4165-8b80-33cd8f78a74b-os-release\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.639291 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1ac35ab-4786-4165-8b80-33cd8f78a74b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.639291 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637727 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-run-netns\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.639291 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637752 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-sysconfig\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.639291 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637874 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e1ac35ab-4786-4165-8b80-33cd8f78a74b-os-release\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.639291 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637897 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-systemd\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.639291 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637958 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-log-socket\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.639291 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.637956 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f07efcef-f35f-45b5-873c-28d5acedb9e0-konnectivity-ca\") pod \"konnectivity-agent-97mzp\" (UID: \"f07efcef-f35f-45b5-873c-28d5acedb9e0\") " pod="kube-system/konnectivity-agent-97mzp" Jul 17 21:46:05.640214 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638021 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-host-var-lib-kubelet\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.640214 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638028 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55dc68af-baa5-4429-a748-6a5ed468c1b3-tmp-dir\") pod \"node-resolver-j6dqx\" (UID: \"55dc68af-baa5-4429-a748-6a5ed468c1b3\") " pod="openshift-dns/node-resolver-j6dqx" Jul 17 21:46:05.640214 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638063 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1ac35ab-4786-4165-8b80-33cd8f78a74b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.640214 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638101 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/55dc68af-baa5-4429-a748-6a5ed468c1b3-hosts-file\") pod \"node-resolver-j6dqx\" (UID: \"55dc68af-baa5-4429-a748-6a5ed468c1b3\") " pod="openshift-dns/node-resolver-j6dqx" Jul 17 21:46:05.640214 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638110 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-host-var-lib-kubelet\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.640214 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/171b015d-bd82-4094-b923-27cddda2aa06-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.640214 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638164 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 17 21:46:05.640214 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638193 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/171b015d-bd82-4094-b923-27cddda2aa06-sys-fs\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.640214 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638200 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1ac35ab-4786-4165-8b80-33cd8f78a74b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.640214 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638223 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-systemd-units\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.640214 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638228 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/171b015d-bd82-4094-b923-27cddda2aa06-kubelet-dir\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.640214 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638226 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1ac35ab-4786-4165-8b80-33cd8f78a74b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.640214 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638257 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/171b015d-bd82-4094-b923-27cddda2aa06-sys-fs\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.640214 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638258 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-slash\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.640214 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638288 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-cnibin\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.640214 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638302 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-hostroot\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.640214 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638318 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1ac35ab-4786-4165-8b80-33cd8f78a74b-cnibin\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.640214 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638333 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/171b015d-bd82-4094-b923-27cddda2aa06-device-dir\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.641174 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638346 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-cnibin\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.641174 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638350 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ad8d984-db0a-4d0b-81cf-7c40926cb7b5-host\") pod \"node-ca-26z9v\" (UID: \"2ad8d984-db0a-4d0b-81cf-7c40926cb7b5\") " pod="openshift-image-registry/node-ca-26z9v" Jul 17 21:46:05.641174 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638363 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-hostroot\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.641174 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638373 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1ac35ab-4786-4165-8b80-33cd8f78a74b-cnibin\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.641174 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638386 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7h22\" (UniqueName: \"kubernetes.io/projected/2ad8d984-db0a-4d0b-81cf-7c40926cb7b5-kube-api-access-d7h22\") pod \"node-ca-26z9v\" (UID: \"2ad8d984-db0a-4d0b-81cf-7c40926cb7b5\") " pod="openshift-image-registry/node-ca-26z9v" Jul 17 21:46:05.641174 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638414 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f038fc3f-ef28-4567-bc5a-0d583093a80e-host-slash\") pod \"iptables-alerter-wxxlz\" (UID: \"f038fc3f-ef28-4567-bc5a-0d583093a80e\") " pod="openshift-network-operator/iptables-alerter-wxxlz" Jul 17 21:46:05.641174 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638421 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/171b015d-bd82-4094-b923-27cddda2aa06-device-dir\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.641174 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638462 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-host-run-k8s-cni-cncf-io\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.641174 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638487 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-os-release\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.641174 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638519 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-host-run-k8s-cni-cncf-io\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.641174 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638528 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-host-var-lib-cni-bin\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.641174 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638561 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cpmq\" (UniqueName: \"kubernetes.io/projected/e1ac35ab-4786-4165-8b80-33cd8f78a74b-kube-api-access-6cpmq\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.641174 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638568 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-host-var-lib-cni-bin\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.641174 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638591 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cb155f63-69cf-4b8c-ad77-324207fefb52-os-release\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.641174 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638597 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-var-lib-openvswitch\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.641174 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-ovnkube-script-lib\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.641174 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638677 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2ad8d984-db0a-4d0b-81cf-7c40926cb7b5-serviceca\") pod \"node-ca-26z9v\" (UID: \"2ad8d984-db0a-4d0b-81cf-7c40926cb7b5\") " pod="openshift-image-registry/node-ca-26z9v" Jul 17 21:46:05.642111 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638706 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-sysctl-d\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.642111 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638735 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/171b015d-bd82-4094-b923-27cddda2aa06-registration-dir\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.642111 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638761 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.642111 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638798 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-env-overrides\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.642111 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638803 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/171b015d-bd82-4094-b923-27cddda2aa06-registration-dir\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.642111 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-sys\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.642111 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.638845 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-lib-modules\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.642111 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.641971 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f07efcef-f35f-45b5-873c-28d5acedb9e0-agent-certs\") pod \"konnectivity-agent-97mzp\" (UID: \"f07efcef-f35f-45b5-873c-28d5acedb9e0\") " pod="kube-system/konnectivity-agent-97mzp" Jul 17 21:46:05.645170 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:05.645151 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:46:05.645275 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:05.645178 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:46:05.645275 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:05.645194 2577 projected.go:196] Error preparing data for projected volume kube-api-access-qngkj for pod openshift-network-diagnostics/network-check-target-kp5bb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:46:05.645275 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:05.645261 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj podName:ee877ecd-672d-4415-a69e-831da8e098fe nodeName:}" failed. No retries permitted until 2026-07-17 21:46:06.145242364 +0000 UTC m=+3.005465859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qngkj" (UniqueName: "kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj") pod "network-check-target-kp5bb" (UID: "ee877ecd-672d-4415-a69e-831da8e098fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:46:05.646500 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.646440 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9744\" (UniqueName: \"kubernetes.io/projected/171b015d-bd82-4094-b923-27cddda2aa06-kube-api-access-c9744\") pod \"aws-ebs-csi-driver-node-w6z6n\" (UID: \"171b015d-bd82-4094-b923-27cddda2aa06\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.646901 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.646856 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxj2n\" (UniqueName: \"kubernetes.io/projected/491a8723-75cd-4309-abad-792b39758916-kube-api-access-sxj2n\") pod \"network-metrics-daemon-4vfnr\" (UID: \"491a8723-75cd-4309-abad-792b39758916\") " pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:05.647268 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.647236 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xsg9\" (UniqueName: \"kubernetes.io/projected/55dc68af-baa5-4429-a748-6a5ed468c1b3-kube-api-access-8xsg9\") pod \"node-resolver-j6dqx\" (UID: \"55dc68af-baa5-4429-a748-6a5ed468c1b3\") " pod="openshift-dns/node-resolver-j6dqx" Jul 17 21:46:05.647711 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.647686 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cpmq\" (UniqueName: \"kubernetes.io/projected/e1ac35ab-4786-4165-8b80-33cd8f78a74b-kube-api-access-6cpmq\") pod \"multus-additional-cni-plugins-fcfh7\" (UID: \"e1ac35ab-4786-4165-8b80-33cd8f78a74b\") " pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.648347 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.648325 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sbtp\" (UniqueName: \"kubernetes.io/projected/cb155f63-69cf-4b8c-ad77-324207fefb52-kube-api-access-8sbtp\") pod \"multus-kdmxg\" (UID: \"cb155f63-69cf-4b8c-ad77-324207fefb52\") " pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.681475 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.681442 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 17 21:46:05.698036 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.697987 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" event={"ID":"30e4b77ac143ec43bfae8d55663eb347","Type":"ContainerStarted","Data":"20a37666ea33792aa309c7808013fb0dcfd786b9d880bed850da8611420e5ac0"} Jul 17 21:46:05.699061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.699040 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" event={"ID":"09f0258f09213a85d398e0f2eb07ae5f","Type":"ContainerStarted","Data":"764a84f37a426c58f82bba91e1dc115aced7622b6923e4b7f28739bc46ce48ab"} Jul 17 21:46:05.740184 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740152 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-cni-bin\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.740349 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740196 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-ovn-node-metrics-cert\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.740349 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740271 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-cni-bin\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.740349 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740327 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2430562-9210-4afd-9800-9e66f9548cb7-dbus\") pod \"global-pull-secret-syncer-982gr\" (UID: \"e2430562-9210-4afd-9800-9e66f9548cb7\") " pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:05.740523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740376 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-run-ovn\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.740523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-ovnkube-config\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.740523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740432 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2430562-9210-4afd-9800-9e66f9548cb7-kubelet-config\") pod \"global-pull-secret-syncer-982gr\" (UID: \"e2430562-9210-4afd-9800-9e66f9548cb7\") " pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:05.740523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740441 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-run-ovn\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.740523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740404 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e2430562-9210-4afd-9800-9e66f9548cb7-dbus\") pod \"global-pull-secret-syncer-982gr\" (UID: \"e2430562-9210-4afd-9800-9e66f9548cb7\") " pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:05.740523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740486 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-kubelet\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.740523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740521 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e2430562-9210-4afd-9800-9e66f9548cb7-kubelet-config\") pod \"global-pull-secret-syncer-982gr\" (UID: \"e2430562-9210-4afd-9800-9e66f9548cb7\") " pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:05.740968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740545 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-kubelet\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.740968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-node-log\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.740968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740612 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxkgh\" (UniqueName: \"kubernetes.io/projected/f038fc3f-ef28-4567-bc5a-0d583093a80e-kube-api-access-dxkgh\") pod \"iptables-alerter-wxxlz\" (UID: \"f038fc3f-ef28-4567-bc5a-0d583093a80e\") " pod="openshift-network-operator/iptables-alerter-wxxlz" Jul 17 21:46:05.740968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740635 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-run-openvswitch\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.740968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740660 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-cni-netd\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.740968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740684 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-sysctl-conf\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.740968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740682 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-node-log\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.740968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740694 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-run-openvswitch\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.740968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740715 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-run-netns\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.740968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740742 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-sysconfig\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.740968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740754 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-cni-netd\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.740968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740765 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-systemd\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.740968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740790 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-log-socket\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.740968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-systemd-units\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.740968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740843 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-slash\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.740968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740860 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-systemd\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.740968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740873 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ad8d984-db0a-4d0b-81cf-7c40926cb7b5-host\") pod \"node-ca-26z9v\" (UID: \"2ad8d984-db0a-4d0b-81cf-7c40926cb7b5\") " pod="openshift-image-registry/node-ca-26z9v" Jul 17 21:46:05.740968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740900 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7h22\" (UniqueName: \"kubernetes.io/projected/2ad8d984-db0a-4d0b-81cf-7c40926cb7b5-kube-api-access-d7h22\") pod \"node-ca-26z9v\" (UID: \"2ad8d984-db0a-4d0b-81cf-7c40926cb7b5\") " pod="openshift-image-registry/node-ca-26z9v" Jul 17 21:46:05.741947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740918 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-sysconfig\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.741947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740923 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-run-netns\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.741947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740925 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f038fc3f-ef28-4567-bc5a-0d583093a80e-host-slash\") pod \"iptables-alerter-wxxlz\" (UID: \"f038fc3f-ef28-4567-bc5a-0d583093a80e\") " pod="openshift-network-operator/iptables-alerter-wxxlz" Jul 17 21:46:05.741947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740973 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-slash\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.741947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.740974 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-var-lib-openvswitch\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.741947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741006 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-var-lib-openvswitch\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.741947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741020 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-ovnkube-script-lib\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.741947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741048 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-systemd-units\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.741947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741050 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2ad8d984-db0a-4d0b-81cf-7c40926cb7b5-serviceca\") pod \"node-ca-26z9v\" (UID: \"2ad8d984-db0a-4d0b-81cf-7c40926cb7b5\") " pod="openshift-image-registry/node-ca-26z9v" Jul 17 21:46:05.741947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741008 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-sysctl-conf\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.741947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741080 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-sysctl-d\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.741947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741113 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.741947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741131 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-log-socket\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.741947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741045 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-ovnkube-config\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.741947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-env-overrides\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.741947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741181 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-sys\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.741947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-lib-modules\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.741947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741226 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-sys\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.743105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741236 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-modprobe-d\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.743105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741181 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.743105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741246 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-sysctl-d\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.743105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741318 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-run\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.743105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741326 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-modprobe-d\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.743105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741350 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-lib-modules\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.743105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741353 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f038fc3f-ef28-4567-bc5a-0d583093a80e-iptables-alerter-script\") pod \"iptables-alerter-wxxlz\" (UID: \"f038fc3f-ef28-4567-bc5a-0d583093a80e\") " pod="openshift-network-operator/iptables-alerter-wxxlz" Jul 17 21:46:05.743105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741367 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2ad8d984-db0a-4d0b-81cf-7c40926cb7b5-host\") pod \"node-ca-26z9v\" (UID: \"2ad8d984-db0a-4d0b-81cf-7c40926cb7b5\") " pod="openshift-image-registry/node-ca-26z9v" Jul 17 21:46:05.743105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741388 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-var-lib-kubelet\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.743105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741400 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-run\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.743105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741406 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f038fc3f-ef28-4567-bc5a-0d583093a80e-host-slash\") pod \"iptables-alerter-wxxlz\" (UID: \"f038fc3f-ef28-4567-bc5a-0d583093a80e\") " pod="openshift-network-operator/iptables-alerter-wxxlz" Jul 17 21:46:05.743105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741409 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2ad8d984-db0a-4d0b-81cf-7c40926cb7b5-serviceca\") pod \"node-ca-26z9v\" (UID: \"2ad8d984-db0a-4d0b-81cf-7c40926cb7b5\") " pod="openshift-image-registry/node-ca-26z9v" Jul 17 21:46:05.743105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-host\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.743105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgd69\" (UniqueName: \"kubernetes.io/projected/f06842a6-0e34-43ba-a98a-ffc1f6501294-kube-api-access-mgd69\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.743105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741461 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-host\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.743105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741463 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-var-lib-kubelet\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.743105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wwpc\" (UniqueName: \"kubernetes.io/projected/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-kube-api-access-2wwpc\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.743105 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741490 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret\") pod \"global-pull-secret-syncer-982gr\" (UID: \"e2430562-9210-4afd-9800-9e66f9548cb7\") " pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:05.744061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741507 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-kubernetes\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.744061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741529 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-tuned\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.744061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741548 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f06842a6-0e34-43ba-a98a-ffc1f6501294-tmp\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.744061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741568 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-env-overrides\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.744061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741564 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-run-ovn-kubernetes\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.744061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741623 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-host-run-ovn-kubernetes\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.744061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741640 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-run-systemd\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.744061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-etc-openvswitch\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.744061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741749 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-run-systemd\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.744061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741756 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-etc-openvswitch\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.744061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.741808 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-kubernetes\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.744061 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:05.741885 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:46:05.744061 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:05.741952 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret podName:e2430562-9210-4afd-9800-9e66f9548cb7 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:06.241934293 +0000 UTC m=+3.102157775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret") pod "global-pull-secret-syncer-982gr" (UID: "e2430562-9210-4afd-9800-9e66f9548cb7") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:46:05.744061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.742047 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-ovnkube-script-lib\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.744061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.742198 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f038fc3f-ef28-4567-bc5a-0d583093a80e-iptables-alerter-script\") pod \"iptables-alerter-wxxlz\" (UID: \"f038fc3f-ef28-4567-bc5a-0d583093a80e\") " pod="openshift-network-operator/iptables-alerter-wxxlz" Jul 17 21:46:05.744061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.743798 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-ovn-node-metrics-cert\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.744934 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.744419 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f06842a6-0e34-43ba-a98a-ffc1f6501294-tmp\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.744992 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.744938 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f06842a6-0e34-43ba-a98a-ffc1f6501294-etc-tuned\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.750274 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.750222 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7h22\" (UniqueName: \"kubernetes.io/projected/2ad8d984-db0a-4d0b-81cf-7c40926cb7b5-kube-api-access-d7h22\") pod \"node-ca-26z9v\" (UID: \"2ad8d984-db0a-4d0b-81cf-7c40926cb7b5\") " pod="openshift-image-registry/node-ca-26z9v" Jul 17 21:46:05.750456 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.750432 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxkgh\" (UniqueName: \"kubernetes.io/projected/f038fc3f-ef28-4567-bc5a-0d583093a80e-kube-api-access-dxkgh\") pod \"iptables-alerter-wxxlz\" (UID: \"f038fc3f-ef28-4567-bc5a-0d583093a80e\") " pod="openshift-network-operator/iptables-alerter-wxxlz" Jul 17 21:46:05.751795 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.751748 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wwpc\" (UniqueName: \"kubernetes.io/projected/f3ab924f-bbc6-42c6-a1bf-8ff0a296966b-kube-api-access-2wwpc\") pod \"ovnkube-node-46wvc\" (UID: \"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b\") " pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:05.751871 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.751810 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgd69\" (UniqueName: \"kubernetes.io/projected/f06842a6-0e34-43ba-a98a-ffc1f6501294-kube-api-access-mgd69\") pod \"tuned-h5xhb\" (UID: \"f06842a6-0e34-43ba-a98a-ffc1f6501294\") " pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.822626 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.822588 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-kdmxg" Jul 17 21:46:05.830002 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.829969 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-97mzp" Jul 17 21:46:05.841426 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.841253 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fcfh7" Jul 17 21:46:05.845432 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.845400 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-j6dqx" Jul 17 21:46:05.853700 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.853679 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" Jul 17 21:46:05.860850 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.860675 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" Jul 17 21:46:05.865760 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.865742 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-26z9v" Jul 17 21:46:05.872885 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.872865 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wxxlz" Jul 17 21:46:05.878253 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:05.878097 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:06.145126 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:06.145041 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs\") pod \"network-metrics-daemon-4vfnr\" (UID: \"491a8723-75cd-4309-abad-792b39758916\") " pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:06.145287 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:06.145167 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:46:06.145287 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:06.145253 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs podName:491a8723-75cd-4309-abad-792b39758916 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:07.145228198 +0000 UTC m=+4.005451693 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs") pod "network-metrics-daemon-4vfnr" (UID: "491a8723-75cd-4309-abad-792b39758916") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:46:06.191621 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:06.191545 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55dc68af_baa5_4429_a748_6a5ed468c1b3.slice/crio-3f7e8e0745234c593d01cd12369bf8405c330aa5d976f3ddff825e57c6cecc78 WatchSource:0}: Error finding container 3f7e8e0745234c593d01cd12369bf8405c330aa5d976f3ddff825e57c6cecc78: Status 404 returned error can't find the container with id 3f7e8e0745234c593d01cd12369bf8405c330aa5d976f3ddff825e57c6cecc78 Jul 17 21:46:06.198788 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:06.198752 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf038fc3f_ef28_4567_bc5a_0d583093a80e.slice/crio-e99bee3603560cf94855bb25529faccb33a32c16247872e95e50b7294db7b4cb WatchSource:0}: Error finding container e99bee3603560cf94855bb25529faccb33a32c16247872e95e50b7294db7b4cb: Status 404 returned error can't find the container with id e99bee3603560cf94855bb25529faccb33a32c16247872e95e50b7294db7b4cb Jul 17 21:46:06.202067 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:06.201800 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171b015d_bd82_4094_b923_27cddda2aa06.slice/crio-b19f9b662b27c8dabcf24fa311e603d478ca26f51de2ae7069681be2fb0a1f6e WatchSource:0}: Error finding container b19f9b662b27c8dabcf24fa311e603d478ca26f51de2ae7069681be2fb0a1f6e: Status 404 returned error can't find the container with id b19f9b662b27c8dabcf24fa311e603d478ca26f51de2ae7069681be2fb0a1f6e Jul 17 21:46:06.203409 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:06.203379 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3ab924f_bbc6_42c6_a1bf_8ff0a296966b.slice/crio-a7bcab3ce1dba068a7c8b973380d65ffccc5ddcd0018dddc603d114cbdb453f7 WatchSource:0}: Error finding container a7bcab3ce1dba068a7c8b973380d65ffccc5ddcd0018dddc603d114cbdb453f7: Status 404 returned error can't find the container with id a7bcab3ce1dba068a7c8b973380d65ffccc5ddcd0018dddc603d114cbdb453f7 Jul 17 21:46:06.205468 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:06.204927 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad8d984_db0a_4d0b_81cf_7c40926cb7b5.slice/crio-cd2e118b5e01076ad849855a99f6f66d00c822af06b1b49663ff399973f81ef6 WatchSource:0}: Error finding container cd2e118b5e01076ad849855a99f6f66d00c822af06b1b49663ff399973f81ef6: Status 404 returned error can't find the container with id cd2e118b5e01076ad849855a99f6f66d00c822af06b1b49663ff399973f81ef6 Jul 17 21:46:06.206625 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:06.206333 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06842a6_0e34_43ba_a98a_ffc1f6501294.slice/crio-a733957901c9d7eef482b7bf1acd5c040e4acfba1651ccc79e659d58e84902dc WatchSource:0}: Error finding container a733957901c9d7eef482b7bf1acd5c040e4acfba1651ccc79e659d58e84902dc: Status 404 returned error can't find the container with id a733957901c9d7eef482b7bf1acd5c040e4acfba1651ccc79e659d58e84902dc Jul 17 21:46:06.245611 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:06.245578 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret\") pod \"global-pull-secret-syncer-982gr\" (UID: \"e2430562-9210-4afd-9800-9e66f9548cb7\") " pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:06.245764 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:06.245656 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qngkj\" (UniqueName: \"kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj\") pod \"network-check-target-kp5bb\" (UID: \"ee877ecd-672d-4415-a69e-831da8e098fe\") " pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:06.245823 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:06.245805 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:46:06.245875 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:06.245834 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:46:06.245875 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:06.245848 2577 projected.go:196] Error preparing data for projected volume kube-api-access-qngkj for pod openshift-network-diagnostics/network-check-target-kp5bb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:46:06.245951 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:06.245899 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj podName:ee877ecd-672d-4415-a69e-831da8e098fe nodeName:}" failed. No retries permitted until 2026-07-17 21:46:07.245881157 +0000 UTC m=+4.106104645 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qngkj" (UniqueName: "kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj") pod "network-check-target-kp5bb" (UID: "ee877ecd-672d-4415-a69e-831da8e098fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:46:06.246011 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:06.245971 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:46:06.246011 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:06.246007 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret podName:e2430562-9210-4afd-9800-9e66f9548cb7 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:07.2459939 +0000 UTC m=+4.106217386 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret") pod "global-pull-secret-syncer-982gr" (UID: "e2430562-9210-4afd-9800-9e66f9548cb7") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:46:06.580008 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:06.579727 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 21:41:05 +0000 UTC" deadline="2027-12-26 17:56:06.101119273 +0000 UTC" Jul 17 21:46:06.580008 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:06.580005 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12644h9m59.521119405s" Jul 17 21:46:06.694867 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:06.694751 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:06.695061 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:06.694887 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:06.702626 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:06.702591 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" event={"ID":"f06842a6-0e34-43ba-a98a-ffc1f6501294","Type":"ContainerStarted","Data":"a733957901c9d7eef482b7bf1acd5c040e4acfba1651ccc79e659d58e84902dc"} Jul 17 21:46:06.704183 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:06.704083 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" event={"ID":"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b","Type":"ContainerStarted","Data":"a7bcab3ce1dba068a7c8b973380d65ffccc5ddcd0018dddc603d114cbdb453f7"} Jul 17 21:46:06.705255 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:06.705226 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-26z9v" event={"ID":"2ad8d984-db0a-4d0b-81cf-7c40926cb7b5","Type":"ContainerStarted","Data":"cd2e118b5e01076ad849855a99f6f66d00c822af06b1b49663ff399973f81ef6"} Jul 17 21:46:06.706335 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:06.706308 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-97mzp" event={"ID":"f07efcef-f35f-45b5-873c-28d5acedb9e0","Type":"ContainerStarted","Data":"afbcc2489b1a87caa83d9cbec87dceb67bacde59d28cfa3a9df0d8ac54f1efe3"} Jul 17 21:46:06.707572 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:06.707524 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fcfh7" event={"ID":"e1ac35ab-4786-4165-8b80-33cd8f78a74b","Type":"ContainerStarted","Data":"faee592f64a2304a9f64037e4ab1e6a7bcb4907259e09176564bfb47dea26a7c"} Jul 17 21:46:06.709632 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:06.709604 2577 generic.go:356] "Generic (PLEG): container finished" podID="30e4b77ac143ec43bfae8d55663eb347" containerID="f57af99ba1cc10aeb4a59c2f446f2f35855bb4fc26e52f61935e4546c43a33e0" exitCode=0 Jul 17 21:46:06.709747 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:06.709638 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" event={"ID":"30e4b77ac143ec43bfae8d55663eb347","Type":"ContainerDied","Data":"f57af99ba1cc10aeb4a59c2f446f2f35855bb4fc26e52f61935e4546c43a33e0"} Jul 17 21:46:06.711068 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:06.710998 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kdmxg" event={"ID":"cb155f63-69cf-4b8c-ad77-324207fefb52","Type":"ContainerStarted","Data":"99057b7338844d2d101d9a30bf770031e88d789dee242cc535ac0c513cd9775a"} Jul 17 21:46:06.712254 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:06.712224 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wxxlz" event={"ID":"f038fc3f-ef28-4567-bc5a-0d583093a80e","Type":"ContainerStarted","Data":"e99bee3603560cf94855bb25529faccb33a32c16247872e95e50b7294db7b4cb"} Jul 17 21:46:06.713459 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:06.713436 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j6dqx" event={"ID":"55dc68af-baa5-4429-a748-6a5ed468c1b3","Type":"ContainerStarted","Data":"3f7e8e0745234c593d01cd12369bf8405c330aa5d976f3ddff825e57c6cecc78"} Jul 17 21:46:06.714364 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:06.714345 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" event={"ID":"171b015d-bd82-4094-b923-27cddda2aa06","Type":"ContainerStarted","Data":"b19f9b662b27c8dabcf24fa311e603d478ca26f51de2ae7069681be2fb0a1f6e"} Jul 17 21:46:07.153015 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:07.152918 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs\") pod \"network-metrics-daemon-4vfnr\" (UID: \"491a8723-75cd-4309-abad-792b39758916\") " pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:07.153193 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:07.153094 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:46:07.153193 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:07.153153 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs podName:491a8723-75cd-4309-abad-792b39758916 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:09.15313443 +0000 UTC m=+6.013357918 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs") pod "network-metrics-daemon-4vfnr" (UID: "491a8723-75cd-4309-abad-792b39758916") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:46:07.254108 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:07.253866 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qngkj\" (UniqueName: \"kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj\") pod \"network-check-target-kp5bb\" (UID: \"ee877ecd-672d-4415-a69e-831da8e098fe\") " pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:07.254108 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:07.253945 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret\") pod \"global-pull-secret-syncer-982gr\" (UID: \"e2430562-9210-4afd-9800-9e66f9548cb7\") " pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:07.254108 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:07.254072 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:46:07.254108 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:07.254104 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:46:07.254108 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:07.254120 2577 projected.go:196] Error preparing data for projected volume kube-api-access-qngkj for pod openshift-network-diagnostics/network-check-target-kp5bb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:46:07.254448 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:07.254079 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:46:07.254448 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:07.254181 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj podName:ee877ecd-672d-4415-a69e-831da8e098fe nodeName:}" failed. No retries permitted until 2026-07-17 21:46:09.254160781 +0000 UTC m=+6.114384268 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qngkj" (UniqueName: "kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj") pod "network-check-target-kp5bb" (UID: "ee877ecd-672d-4415-a69e-831da8e098fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:46:07.254448 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:07.254246 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret podName:e2430562-9210-4afd-9800-9e66f9548cb7 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:09.254214793 +0000 UTC m=+6.114438279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret") pod "global-pull-secret-syncer-982gr" (UID: "e2430562-9210-4afd-9800-9e66f9548cb7") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:46:07.581223 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:07.581177 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 21:41:05 +0000 UTC" deadline="2028-02-23 07:12:31.313564891 +0000 UTC" Jul 17 21:46:07.581223 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:07.581220 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14049h26m23.732350354s" Jul 17 21:46:07.698854 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:07.698825 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:07.699032 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:07.698963 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:07.699410 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:07.699390 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:07.699583 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:07.699562 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:07.740304 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:07.739644 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" event={"ID":"30e4b77ac143ec43bfae8d55663eb347","Type":"ContainerStarted","Data":"5c31a3ad154d6ea2fba95ab4f83539f1b42840e8fdbb43e97a005b2e784072d1"} Jul 17 21:46:07.754567 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:07.754534 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" event={"ID":"09f0258f09213a85d398e0f2eb07ae5f","Type":"ContainerStarted","Data":"f3f1c19f1f99afe84c587d14e2e5a18bb455ddf24510d6532f94e44ab8b2595a"} Jul 17 21:46:07.771625 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:07.771455 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-14.ec2.internal" podStartSLOduration=2.771435502 podStartE2EDuration="2.771435502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:07.770559642 +0000 UTC m=+4.630783168" watchObservedRunningTime="2026-07-17 21:46:07.771435502 +0000 UTC m=+4.631659005" Jul 17 21:46:07.771765 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:07.771645 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-14.ec2.internal" podStartSLOduration=2.7716352239999997 podStartE2EDuration="2.771635224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:07.756784679 +0000 UTC m=+4.617008206" watchObservedRunningTime="2026-07-17 21:46:07.771635224 +0000 UTC m=+4.631858729" Jul 17 21:46:08.693953 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:08.693919 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:08.694457 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:08.694052 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:09.173993 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:09.173691 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs\") pod \"network-metrics-daemon-4vfnr\" (UID: \"491a8723-75cd-4309-abad-792b39758916\") " pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:09.173993 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:09.173847 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:46:09.173993 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:09.173905 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs podName:491a8723-75cd-4309-abad-792b39758916 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:13.173888001 +0000 UTC m=+10.034111488 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs") pod "network-metrics-daemon-4vfnr" (UID: "491a8723-75cd-4309-abad-792b39758916") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:46:09.274880 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:09.274842 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret\") pod \"global-pull-secret-syncer-982gr\" (UID: \"e2430562-9210-4afd-9800-9e66f9548cb7\") " pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:09.275104 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:09.274918 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qngkj\" (UniqueName: \"kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj\") pod \"network-check-target-kp5bb\" (UID: \"ee877ecd-672d-4415-a69e-831da8e098fe\") " pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:09.275104 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:09.275058 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:46:09.275104 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:09.275076 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:46:09.275104 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:09.275088 2577 projected.go:196] Error preparing data for projected volume kube-api-access-qngkj for pod openshift-network-diagnostics/network-check-target-kp5bb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:46:09.275318 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:09.275145 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj podName:ee877ecd-672d-4415-a69e-831da8e098fe nodeName:}" failed. No retries permitted until 2026-07-17 21:46:13.275127965 +0000 UTC m=+10.135351444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qngkj" (UniqueName: "kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj") pod "network-check-target-kp5bb" (UID: "ee877ecd-672d-4415-a69e-831da8e098fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:46:09.275575 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:09.275552 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:46:09.275678 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:09.275609 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret podName:e2430562-9210-4afd-9800-9e66f9548cb7 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:13.275593902 +0000 UTC m=+10.135817386 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret") pod "global-pull-secret-syncer-982gr" (UID: "e2430562-9210-4afd-9800-9e66f9548cb7") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:46:09.695282 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:09.694716 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:09.695282 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:09.694729 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:09.695282 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:09.694846 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:09.695282 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:09.694999 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:10.693875 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:10.693848 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:10.694080 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:10.693966 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:11.696111 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:11.695617 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:11.696111 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:11.695761 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:11.696639 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:11.696227 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:11.696639 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:11.696337 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:12.694364 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:12.694331 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:12.694570 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:12.694449 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:13.211189 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:13.211155 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs\") pod \"network-metrics-daemon-4vfnr\" (UID: \"491a8723-75cd-4309-abad-792b39758916\") " pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:13.211694 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:13.211332 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:46:13.211694 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:13.211397 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs podName:491a8723-75cd-4309-abad-792b39758916 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:21.211381692 +0000 UTC m=+18.071605175 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs") pod "network-metrics-daemon-4vfnr" (UID: "491a8723-75cd-4309-abad-792b39758916") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:46:13.312446 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:13.312407 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret\") pod \"global-pull-secret-syncer-982gr\" (UID: \"e2430562-9210-4afd-9800-9e66f9548cb7\") " pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:13.312644 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:13.312504 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qngkj\" (UniqueName: \"kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj\") pod \"network-check-target-kp5bb\" (UID: \"ee877ecd-672d-4415-a69e-831da8e098fe\") " pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:13.312718 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:13.312669 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:46:13.312718 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:13.312690 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:46:13.312718 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:13.312699 2577 projected.go:196] Error preparing data for projected volume kube-api-access-qngkj for pod openshift-network-diagnostics/network-check-target-kp5bb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:46:13.312872 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:13.312757 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj podName:ee877ecd-672d-4415-a69e-831da8e098fe nodeName:}" failed. No retries permitted until 2026-07-17 21:46:21.312737671 +0000 UTC m=+18.172961167 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qngkj" (UniqueName: "kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj") pod "network-check-target-kp5bb" (UID: "ee877ecd-672d-4415-a69e-831da8e098fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:46:13.313133 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:13.313115 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:46:13.313204 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:13.313167 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret podName:e2430562-9210-4afd-9800-9e66f9548cb7 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:21.313153007 +0000 UTC m=+18.173376490 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret") pod "global-pull-secret-syncer-982gr" (UID: "e2430562-9210-4afd-9800-9e66f9548cb7") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:46:13.696663 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:13.696625 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:13.696829 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:13.696768 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:13.698502 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:13.698463 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:13.698626 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:13.698600 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:14.694719 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:14.694556 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:14.694719 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:14.694686 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:15.694608 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:15.694575 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:15.694778 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:15.694707 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:15.696519 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:15.696361 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:15.696519 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:15.696452 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:16.694542 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:16.694478 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:16.694717 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:16.694651 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:17.696731 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:17.696697 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:17.697138 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:17.696827 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:17.697138 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:17.696894 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:17.697138 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:17.696949 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:18.693795 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:18.693754 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:18.693980 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:18.693867 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:19.694431 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:19.694400 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:19.694904 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:19.694413 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:19.694904 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:19.694548 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:19.694904 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:19.694607 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:20.693927 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:20.693898 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:20.694124 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:20.694009 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:21.273167 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:21.273136 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs\") pod \"network-metrics-daemon-4vfnr\" (UID: \"491a8723-75cd-4309-abad-792b39758916\") " pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:21.273767 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:21.273267 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:46:21.273767 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:21.273332 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs podName:491a8723-75cd-4309-abad-792b39758916 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:37.273317308 +0000 UTC m=+34.133540788 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs") pod "network-metrics-daemon-4vfnr" (UID: "491a8723-75cd-4309-abad-792b39758916") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:46:21.373725 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:21.373687 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qngkj\" (UniqueName: \"kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj\") pod \"network-check-target-kp5bb\" (UID: \"ee877ecd-672d-4415-a69e-831da8e098fe\") " pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:21.373908 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:21.373767 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret\") pod \"global-pull-secret-syncer-982gr\" (UID: \"e2430562-9210-4afd-9800-9e66f9548cb7\") " pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:21.373908 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:21.373845 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:46:21.373908 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:21.373871 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:46:21.373908 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:21.373884 2577 projected.go:196] Error preparing data for projected volume kube-api-access-qngkj for pod openshift-network-diagnostics/network-check-target-kp5bb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:46:21.373908 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:21.373883 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:46:21.374112 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:21.373949 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret podName:e2430562-9210-4afd-9800-9e66f9548cb7 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:37.373932165 +0000 UTC m=+34.234155647 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret") pod "global-pull-secret-syncer-982gr" (UID: "e2430562-9210-4afd-9800-9e66f9548cb7") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:46:21.374112 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:21.373971 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj podName:ee877ecd-672d-4415-a69e-831da8e098fe nodeName:}" failed. No retries permitted until 2026-07-17 21:46:37.373960306 +0000 UTC m=+34.234183789 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qngkj" (UniqueName: "kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj") pod "network-check-target-kp5bb" (UID: "ee877ecd-672d-4415-a69e-831da8e098fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:46:21.694552 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:21.694520 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:21.694552 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:21.694553 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:21.694803 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:21.694660 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:21.694803 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:21.694778 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:22.694503 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:22.694456 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:22.694917 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:22.694600 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:23.695543 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:23.695505 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:23.695979 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:23.695627 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:23.695979 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:23.695687 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:23.695979 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:23.695830 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:24.694609 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:24.694575 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:24.694786 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:24.694711 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:25.694844 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:25.694809 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:25.695244 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:25.694924 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:25.695244 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:25.694935 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:25.695244 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:25.695069 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:26.695042 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:26.694656 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:26.695805 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:26.695158 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:26.794017 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:26.793990 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" event={"ID":"171b015d-bd82-4094-b923-27cddda2aa06","Type":"ContainerStarted","Data":"7f4987861eca1ec6de1c8f5bbe0470b6d177412a7039744b15ebe3159f2975da"} Jul 17 21:46:26.795247 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:26.795228 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" event={"ID":"f06842a6-0e34-43ba-a98a-ffc1f6501294","Type":"ContainerStarted","Data":"a7f5ad26676495e003eee644531554ba519b422357ba725e872bd54feb5257a8"} Jul 17 21:46:26.797563 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:26.797535 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" event={"ID":"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b","Type":"ContainerStarted","Data":"c3133439a4a01ae8a33385488831e273c647fa010ea7ea99b7eae65f684a0893"} Jul 17 21:46:26.797665 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:26.797594 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" event={"ID":"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b","Type":"ContainerStarted","Data":"81b265928da01543dbbdd7745c47de9c20d29014bed42033e9bdc58bb543ee8f"} Jul 17 21:46:26.797665 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:26.797632 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" event={"ID":"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b","Type":"ContainerStarted","Data":"ebb35ed4f53e319e15a4b97e29e17a13e70591d26388c643760ee4bf31770a83"} Jul 17 21:46:26.798885 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:26.798866 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-26z9v" event={"ID":"2ad8d984-db0a-4d0b-81cf-7c40926cb7b5","Type":"ContainerStarted","Data":"8ef9fc6b220fc0cf231c54a871ddc4a1166d16dd92c187a5391d5d22ebe554df"} Jul 17 21:46:26.800335 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:26.800314 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-97mzp" event={"ID":"f07efcef-f35f-45b5-873c-28d5acedb9e0","Type":"ContainerStarted","Data":"46c6fb4e7bc3f91d85fba5821fff76515a5577c72e545a49085b50315fc9f573"} Jul 17 21:46:26.801836 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:26.801804 2577 generic.go:356] "Generic (PLEG): container finished" podID="e1ac35ab-4786-4165-8b80-33cd8f78a74b" containerID="3e53b0289e90863b2d7539b3a7d1f7edf3e1869db2e168e71990a673b04504fb" exitCode=0 Jul 17 21:46:26.801925 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:26.801833 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fcfh7" event={"ID":"e1ac35ab-4786-4165-8b80-33cd8f78a74b","Type":"ContainerDied","Data":"3e53b0289e90863b2d7539b3a7d1f7edf3e1869db2e168e71990a673b04504fb"} Jul 17 21:46:26.804323 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:26.804281 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-kdmxg" event={"ID":"cb155f63-69cf-4b8c-ad77-324207fefb52","Type":"ContainerStarted","Data":"fc808443a709cc80ee01cfc5341465c3ddc916a3317eb936e48db7b2f32bb71b"} Jul 17 21:46:26.806137 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:26.806114 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-j6dqx" event={"ID":"55dc68af-baa5-4429-a748-6a5ed468c1b3","Type":"ContainerStarted","Data":"15cf311ecbe9948c18f1f3f04c01d145278ab8e441763bf8f09324b4b8191082"} Jul 17 21:46:26.812608 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:26.812570 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h5xhb" podStartSLOduration=3.204141403 podStartE2EDuration="22.812558486s" totalImagesPullingTime="19.608417083s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:26.812215837 +0000 UTC m=+23.672439338" watchObservedRunningTime="2026-07-17 21:46:26.812558486 +0000 UTC m=+23.672781989" Jul 17 21:46:26.827062 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:26.827023 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-97mzp" podStartSLOduration=3.2959835650000002 podStartE2EDuration="22.827012309s" totalImagesPullingTime="19.531028744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:26.826834269 +0000 UTC m=+23.687057771" watchObservedRunningTime="2026-07-17 21:46:26.827012309 +0000 UTC m=+23.687235810" Jul 17 21:46:26.861034 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:26.860988 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-kdmxg" podStartSLOduration=4.24669963 podStartE2EDuration="23.860973806s" totalImagesPullingTime="19.614274176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:26.860518159 +0000 UTC m=+23.720741680" watchObservedRunningTime="2026-07-17 21:46:26.860973806 +0000 UTC m=+23.721197307" Jul 17 21:46:26.873825 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:26.873785 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-j6dqx" podStartSLOduration=3.247829147 podStartE2EDuration="22.87377166s" totalImagesPullingTime="19.625942513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:26.873273634 +0000 UTC m=+23.733497136" watchObservedRunningTime="2026-07-17 21:46:26.87377166 +0000 UTC m=+23.733995161" Jul 17 21:46:27.158075 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:27.158046 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 17 21:46:27.587515 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:27.587401 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T21:46:27.158068047Z","UUID":"a83fae7e-ab5f-40b2-86b6-ed86d07c639f","Handler":null,"Name":"","Endpoint":""} Jul 17 21:46:27.590010 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:27.589991 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 17 21:46:27.590126 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:27.590025 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 17 21:46:27.697343 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:27.697319 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:27.697799 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:27.697444 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:27.697799 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:27.697511 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:27.697799 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:27.697596 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:27.810398 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:27.810361 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" event={"ID":"171b015d-bd82-4094-b923-27cddda2aa06","Type":"ContainerStarted","Data":"3e3ed346477d5085d050db5c98bca7bfbad6708cd0a7b09537ecb2457854dc80"} Jul 17 21:46:27.814355 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:27.814326 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" event={"ID":"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b","Type":"ContainerStarted","Data":"a67193267c93193ea4115b49674bbb1b1e6f5bfc1c2c988e2aec681d5cd07962"} Jul 17 21:46:27.814544 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:27.814407 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" event={"ID":"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b","Type":"ContainerStarted","Data":"2bb9ddbd623defe67948a8c4d26fba133b22c3912a5e6dbef04044760c9bb971"} Jul 17 21:46:27.814544 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:27.814456 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" event={"ID":"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b","Type":"ContainerStarted","Data":"ce9336aa35cfa9805daf8606bf361f74ba60f1af567f6d63f2899be44b3a4781"} Jul 17 21:46:27.815968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:27.815944 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wxxlz" event={"ID":"f038fc3f-ef28-4567-bc5a-0d583093a80e","Type":"ContainerStarted","Data":"097a325adba3551b6af60189e811ebc1e46611a7379b08ae18718288b7cf768c"} Jul 17 21:46:27.830260 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:27.830201 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-26z9v" podStartSLOduration=4.221154095 podStartE2EDuration="23.830183554s" totalImagesPullingTime="19.609029459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:26.88572288 +0000 UTC m=+23.745946386" watchObservedRunningTime="2026-07-17 21:46:27.830183554 +0000 UTC m=+24.690407058" Jul 17 21:46:27.830418 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:27.830304 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wxxlz" podStartSLOduration=4.311815812 podStartE2EDuration="23.830296728s" totalImagesPullingTime="19.518480916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:27.829614817 +0000 UTC m=+24.689838320" watchObservedRunningTime="2026-07-17 21:46:27.830296728 +0000 UTC m=+24.690520230" Jul 17 21:46:28.694358 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:28.694333 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:28.694571 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:28.694426 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:28.822135 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:28.822097 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" event={"ID":"171b015d-bd82-4094-b923-27cddda2aa06","Type":"ContainerStarted","Data":"acd0145e6bb57f57337267b2283ae49714e99eb9b2bffc6e78dd729d3c007c8c"} Jul 17 21:46:28.841531 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:28.841464 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-w6z6n" podStartSLOduration=3.465264636 podStartE2EDuration="24.841446419s" totalImagesPullingTime="21.376181783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:46:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:28.840375494 +0000 UTC m=+25.700598998" watchObservedRunningTime="2026-07-17 21:46:28.841446419 +0000 UTC m=+25.701669921" Jul 17 21:46:28.997026 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:28.996954 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-97mzp" Jul 17 21:46:29.697466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:29.697285 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:29.697634 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:29.697299 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:29.697634 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:29.697596 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:29.697838 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:29.697790 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:29.827516 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:29.827473 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" event={"ID":"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b","Type":"ContainerStarted","Data":"318146c66e18dcd3103ba5d8188a69810d566d1c732fef9c2ae6dbb1f3f0733c"} Jul 17 21:46:30.693756 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:30.693720 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:30.693979 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:30.693842 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:31.225642 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:31.225609 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-97mzp" Jul 17 21:46:31.226236 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:31.226219 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-97mzp" Jul 17 21:46:31.694452 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:31.694422 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:31.694624 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:31.694561 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:31.694692 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:31.694619 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:31.694762 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:31.694737 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:31.835066 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:31.834824 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" event={"ID":"f3ab924f-bbc6-42c6-a1bf-8ff0a296966b","Type":"ContainerStarted","Data":"0c9bf7d58fdd38cba51d989975cc28ee7ab52a74168c390396ed6c0b5eeb1277"} Jul 17 21:46:31.835174 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:31.835140 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:31.835235 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:31.835195 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:31.835235 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:31.835226 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:31.837029 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:31.837004 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fcfh7" event={"ID":"e1ac35ab-4786-4165-8b80-33cd8f78a74b","Type":"ContainerStarted","Data":"8b172599f59db53254e8d05bf2d729eb50ce188a988c2ba50f78430d27b9bc3a"} Jul 17 21:46:31.837712 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:31.837694 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-97mzp" Jul 17 21:46:31.853829 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:31.853809 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:31.859410 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:31.859369 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" podStartSLOduration=7.929136475 podStartE2EDuration="27.85935544s" totalImagesPullingTime="19.930218965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:31.858603552 +0000 UTC m=+28.718827074" watchObservedRunningTime="2026-07-17 21:46:31.85935544 +0000 UTC m=+28.719578942" Jul 17 21:46:31.865701 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:31.865681 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:46:32.694668 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:32.694642 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:32.695405 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:32.694729 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:32.840817 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:32.840782 2577 generic.go:356] "Generic (PLEG): container finished" podID="e1ac35ab-4786-4165-8b80-33cd8f78a74b" containerID="8b172599f59db53254e8d05bf2d729eb50ce188a988c2ba50f78430d27b9bc3a" exitCode=0 Jul 17 21:46:32.840994 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:32.840863 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fcfh7" event={"ID":"e1ac35ab-4786-4165-8b80-33cd8f78a74b","Type":"ContainerDied","Data":"8b172599f59db53254e8d05bf2d729eb50ce188a988c2ba50f78430d27b9bc3a"} Jul 17 21:46:33.695922 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:33.695884 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:33.696463 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:33.696009 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:33.696463 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:33.696053 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:33.696463 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:33.696141 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:34.694318 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:34.694290 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:34.694522 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:34.694382 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:34.847089 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:34.847051 2577 generic.go:356] "Generic (PLEG): container finished" podID="e1ac35ab-4786-4165-8b80-33cd8f78a74b" containerID="9546664cb2ef5a7cf91740af819a72f9a3afaba0104a82514898c415fc46d565" exitCode=0 Jul 17 21:46:34.847089 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:34.847089 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fcfh7" event={"ID":"e1ac35ab-4786-4165-8b80-33cd8f78a74b","Type":"ContainerDied","Data":"9546664cb2ef5a7cf91740af819a72f9a3afaba0104a82514898c415fc46d565"} Jul 17 21:46:35.694096 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:35.694066 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:35.694273 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:35.694180 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:35.694318 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:35.694278 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:35.694383 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:35.694364 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:36.694755 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:36.694717 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:36.695142 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:36.694820 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:36.853286 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:36.853256 2577 generic.go:356] "Generic (PLEG): container finished" podID="e1ac35ab-4786-4165-8b80-33cd8f78a74b" containerID="17f459b4d3ced494ad2218368c266004a558e3cc2f6141d51ede84de79da8ad4" exitCode=0 Jul 17 21:46:36.853442 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:36.853303 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fcfh7" event={"ID":"e1ac35ab-4786-4165-8b80-33cd8f78a74b","Type":"ContainerDied","Data":"17f459b4d3ced494ad2218368c266004a558e3cc2f6141d51ede84de79da8ad4"} Jul 17 21:46:37.294091 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:37.294057 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs\") pod \"network-metrics-daemon-4vfnr\" (UID: \"491a8723-75cd-4309-abad-792b39758916\") " pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:37.294257 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:37.294174 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:46:37.294257 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:37.294223 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs podName:491a8723-75cd-4309-abad-792b39758916 nodeName:}" failed. No retries permitted until 2026-07-17 21:47:09.294210104 +0000 UTC m=+66.154433585 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs") pod "network-metrics-daemon-4vfnr" (UID: "491a8723-75cd-4309-abad-792b39758916") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 17 21:46:37.394922 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:37.394888 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret\") pod \"global-pull-secret-syncer-982gr\" (UID: \"e2430562-9210-4afd-9800-9e66f9548cb7\") " pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:37.395090 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:37.394943 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qngkj\" (UniqueName: \"kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj\") pod \"network-check-target-kp5bb\" (UID: \"ee877ecd-672d-4415-a69e-831da8e098fe\") " pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:37.395090 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:37.395040 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 17 21:46:37.395090 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:37.395058 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 17 21:46:37.395090 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:37.395073 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 17 21:46:37.395090 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:37.395083 2577 projected.go:196] Error preparing data for projected volume kube-api-access-qngkj for pod openshift-network-diagnostics/network-check-target-kp5bb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:46:37.395345 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:37.395112 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret podName:e2430562-9210-4afd-9800-9e66f9548cb7 nodeName:}" failed. No retries permitted until 2026-07-17 21:47:09.395092473 +0000 UTC m=+66.255315954 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret") pod "global-pull-secret-syncer-982gr" (UID: "e2430562-9210-4afd-9800-9e66f9548cb7") : object "kube-system"/"original-pull-secret" not registered Jul 17 21:46:37.395345 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:37.395129 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj podName:ee877ecd-672d-4415-a69e-831da8e098fe nodeName:}" failed. No retries permitted until 2026-07-17 21:47:09.395122688 +0000 UTC m=+66.255346168 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qngkj" (UniqueName: "kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj") pod "network-check-target-kp5bb" (UID: "ee877ecd-672d-4415-a69e-831da8e098fe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 17 21:46:37.695180 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:37.694660 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:37.695180 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:37.694706 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:37.695180 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:37.694819 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:37.695180 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:37.694940 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:38.694358 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:38.694321 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:38.694541 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:38.694452 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:39.693871 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:39.693839 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:39.694311 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:39.693966 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:39.694311 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:39.694023 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:39.694311 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:39.694157 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:40.694742 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:40.694704 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:40.695164 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:40.694825 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:41.606984 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:41.606766 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-982gr"] Jul 17 21:46:41.607116 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:41.607003 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:41.607116 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:41.607079 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:41.609420 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:41.609398 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kp5bb"] Jul 17 21:46:41.609536 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:41.609471 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:41.609593 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:41.609570 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:41.609864 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:41.609847 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4vfnr"] Jul 17 21:46:41.609929 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:41.609920 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:41.610010 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:41.609994 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:41.869008 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:41.868932 2577 generic.go:356] "Generic (PLEG): container finished" podID="e1ac35ab-4786-4165-8b80-33cd8f78a74b" containerID="114ca234d2862e24d7d9eb85a1f68358a041fdfb9c2640dbfe259a0caa80a267" exitCode=0 Jul 17 21:46:41.869008 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:41.868982 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fcfh7" event={"ID":"e1ac35ab-4786-4165-8b80-33cd8f78a74b","Type":"ContainerDied","Data":"114ca234d2862e24d7d9eb85a1f68358a041fdfb9c2640dbfe259a0caa80a267"} Jul 17 21:46:42.874267 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:42.874236 2577 generic.go:356] "Generic (PLEG): container finished" podID="e1ac35ab-4786-4165-8b80-33cd8f78a74b" containerID="ff9f9d524376da436acd6b4200a23065505a6e32e610d7d1c64f9ba837228b32" exitCode=0 Jul 17 21:46:42.874887 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:42.874305 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fcfh7" event={"ID":"e1ac35ab-4786-4165-8b80-33cd8f78a74b","Type":"ContainerDied","Data":"ff9f9d524376da436acd6b4200a23065505a6e32e610d7d1c64f9ba837228b32"} Jul 17 21:46:43.695451 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:43.695417 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:43.695631 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:43.695517 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:43.695631 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:43.695557 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:43.695744 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:43.695668 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:43.695744 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:43.695710 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:43.695845 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:43.695800 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:43.879540 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:43.879510 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fcfh7" event={"ID":"e1ac35ab-4786-4165-8b80-33cd8f78a74b","Type":"ContainerStarted","Data":"7eb9508148112aea846115effe55f6401ffcbb67e49af99bf898e5b1837ef122"} Jul 17 21:46:43.901809 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:43.901764 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fcfh7" podStartSLOduration=9.021784719 podStartE2EDuration="39.901748823s" totalImagesPullingTime="30.879964104s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:46:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:43.901308288 +0000 UTC m=+40.761531789" watchObservedRunningTime="2026-07-17 21:46:43.901748823 +0000 UTC m=+40.761972325" Jul 17 21:46:45.693891 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:45.693862 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:45.694306 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:45.693862 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:45.694306 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:45.693990 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:45.694306 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:45.693988 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4vfnr" podUID="491a8723-75cd-4309-abad-792b39758916" Jul 17 21:46:45.694306 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:45.694070 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-982gr" podUID="e2430562-9210-4afd-9800-9e66f9548cb7" Jul 17 21:46:45.694306 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:45.694154 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kp5bb" podUID="ee877ecd-672d-4415-a69e-831da8e098fe" Jul 17 21:46:45.946691 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:45.946619 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-14.ec2.internal" event="NodeReady" Jul 17 21:46:45.946827 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:45.946756 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 17 21:46:46.005764 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.005736 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ntxm7"] Jul 17 21:46:46.026135 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.026112 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4vnfl"] Jul 17 21:46:46.026261 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.026237 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ntxm7" Jul 17 21:46:46.028630 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.028609 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 17 21:46:46.028776 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.028649 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 17 21:46:46.028776 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.028718 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 17 21:46:46.028891 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.028792 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 17 21:46:46.029025 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.029008 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9pf4b\"" Jul 17 21:46:46.042271 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.042252 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ntxm7"] Jul 17 21:46:46.042353 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.042303 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4vnfl"] Jul 17 21:46:46.042353 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.042255 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4vnfl" Jul 17 21:46:46.044931 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.044911 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 17 21:46:46.044931 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.044915 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-q82nm\"" Jul 17 21:46:46.045264 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.045246 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 17 21:46:46.109010 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.108978 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4pwvp"] Jul 17 21:46:46.134149 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.134126 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4pwvp"] Jul 17 21:46:46.134248 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.134180 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4pwvp" Jul 17 21:46:46.137329 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.137308 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-l5qwj\"" Jul 17 21:46:46.137448 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.137431 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 17 21:46:46.137548 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.137532 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 17 21:46:46.138268 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.138254 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 17 21:46:46.163431 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.163408 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aac16a59-a485-4da6-8093-78fb71a06bcd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntxm7\" (UID: \"aac16a59-a485-4da6-8093-78fb71a06bcd\") " pod="openshift-insights/insights-runtime-extractor-ntxm7" Jul 17 21:46:46.163556 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.163443 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed-config-volume\") pod \"dns-default-4vnfl\" (UID: \"e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed\") " pod="openshift-dns/dns-default-4vnfl" Jul 17 21:46:46.163556 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.163468 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aac16a59-a485-4da6-8093-78fb71a06bcd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntxm7\" (UID: \"aac16a59-a485-4da6-8093-78fb71a06bcd\") " pod="openshift-insights/insights-runtime-extractor-ntxm7" Jul 17 21:46:46.163556 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.163537 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aac16a59-a485-4da6-8093-78fb71a06bcd-data-volume\") pod \"insights-runtime-extractor-ntxm7\" (UID: \"aac16a59-a485-4da6-8093-78fb71a06bcd\") " pod="openshift-insights/insights-runtime-extractor-ntxm7" Jul 17 21:46:46.163701 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.163570 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r9t5\" (UniqueName: \"kubernetes.io/projected/aac16a59-a485-4da6-8093-78fb71a06bcd-kube-api-access-7r9t5\") pod \"insights-runtime-extractor-ntxm7\" (UID: \"aac16a59-a485-4da6-8093-78fb71a06bcd\") " pod="openshift-insights/insights-runtime-extractor-ntxm7" Jul 17 21:46:46.163701 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.163621 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed-tmp-dir\") pod \"dns-default-4vnfl\" (UID: \"e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed\") " pod="openshift-dns/dns-default-4vnfl" Jul 17 21:46:46.163701 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.163639 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcmnm\" (UniqueName: \"kubernetes.io/projected/e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed-kube-api-access-rcmnm\") pod \"dns-default-4vnfl\" (UID: \"e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed\") " pod="openshift-dns/dns-default-4vnfl" Jul 17 21:46:46.163701 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.163680 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aac16a59-a485-4da6-8093-78fb71a06bcd-crio-socket\") pod \"insights-runtime-extractor-ntxm7\" (UID: \"aac16a59-a485-4da6-8093-78fb71a06bcd\") " pod="openshift-insights/insights-runtime-extractor-ntxm7" Jul 17 21:46:46.163839 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.163705 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed-metrics-tls\") pod \"dns-default-4vnfl\" (UID: \"e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed\") " pod="openshift-dns/dns-default-4vnfl" Jul 17 21:46:46.264921 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.264848 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aac16a59-a485-4da6-8093-78fb71a06bcd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntxm7\" (UID: \"aac16a59-a485-4da6-8093-78fb71a06bcd\") " pod="openshift-insights/insights-runtime-extractor-ntxm7" Jul 17 21:46:46.264921 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.264884 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67344770-d2f2-44bb-a026-130469fe7114-cert\") pod \"ingress-canary-4pwvp\" (UID: \"67344770-d2f2-44bb-a026-130469fe7114\") " pod="openshift-ingress-canary/ingress-canary-4pwvp" Jul 17 21:46:46.264921 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.264903 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed-config-volume\") pod \"dns-default-4vnfl\" (UID: \"e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed\") " pod="openshift-dns/dns-default-4vnfl" Jul 17 21:46:46.264921 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.264920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aac16a59-a485-4da6-8093-78fb71a06bcd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntxm7\" (UID: \"aac16a59-a485-4da6-8093-78fb71a06bcd\") " pod="openshift-insights/insights-runtime-extractor-ntxm7" Jul 17 21:46:46.265211 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.264967 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aac16a59-a485-4da6-8093-78fb71a06bcd-data-volume\") pod \"insights-runtime-extractor-ntxm7\" (UID: \"aac16a59-a485-4da6-8093-78fb71a06bcd\") " pod="openshift-insights/insights-runtime-extractor-ntxm7" Jul 17 21:46:46.265211 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.265010 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r9t5\" (UniqueName: \"kubernetes.io/projected/aac16a59-a485-4da6-8093-78fb71a06bcd-kube-api-access-7r9t5\") pod \"insights-runtime-extractor-ntxm7\" (UID: \"aac16a59-a485-4da6-8093-78fb71a06bcd\") " pod="openshift-insights/insights-runtime-extractor-ntxm7" Jul 17 21:46:46.265211 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.265168 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed-tmp-dir\") pod \"dns-default-4vnfl\" (UID: \"e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed\") " pod="openshift-dns/dns-default-4vnfl" Jul 17 21:46:46.265211 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.265196 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcmnm\" (UniqueName: \"kubernetes.io/projected/e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed-kube-api-access-rcmnm\") pod \"dns-default-4vnfl\" (UID: \"e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed\") " pod="openshift-dns/dns-default-4vnfl" Jul 17 21:46:46.265411 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.265284 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aac16a59-a485-4da6-8093-78fb71a06bcd-crio-socket\") pod \"insights-runtime-extractor-ntxm7\" (UID: \"aac16a59-a485-4da6-8093-78fb71a06bcd\") " pod="openshift-insights/insights-runtime-extractor-ntxm7" Jul 17 21:46:46.265411 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.265334 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm9cz\" (UniqueName: \"kubernetes.io/projected/67344770-d2f2-44bb-a026-130469fe7114-kube-api-access-xm9cz\") pod \"ingress-canary-4pwvp\" (UID: \"67344770-d2f2-44bb-a026-130469fe7114\") " pod="openshift-ingress-canary/ingress-canary-4pwvp" Jul 17 21:46:46.265411 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.265360 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed-metrics-tls\") pod \"dns-default-4vnfl\" (UID: \"e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed\") " pod="openshift-dns/dns-default-4vnfl" Jul 17 21:46:46.265587 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.265441 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed-tmp-dir\") pod \"dns-default-4vnfl\" (UID: \"e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed\") " pod="openshift-dns/dns-default-4vnfl" Jul 17 21:46:46.265652 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.265590 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed-config-volume\") pod \"dns-default-4vnfl\" (UID: \"e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed\") " pod="openshift-dns/dns-default-4vnfl" Jul 17 21:46:46.265652 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.265630 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/aac16a59-a485-4da6-8093-78fb71a06bcd-crio-socket\") pod \"insights-runtime-extractor-ntxm7\" (UID: \"aac16a59-a485-4da6-8093-78fb71a06bcd\") " pod="openshift-insights/insights-runtime-extractor-ntxm7" Jul 17 21:46:46.265741 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.265659 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/aac16a59-a485-4da6-8093-78fb71a06bcd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ntxm7\" (UID: \"aac16a59-a485-4da6-8093-78fb71a06bcd\") " pod="openshift-insights/insights-runtime-extractor-ntxm7" Jul 17 21:46:46.265879 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.265859 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/aac16a59-a485-4da6-8093-78fb71a06bcd-data-volume\") pod \"insights-runtime-extractor-ntxm7\" (UID: \"aac16a59-a485-4da6-8093-78fb71a06bcd\") " pod="openshift-insights/insights-runtime-extractor-ntxm7" Jul 17 21:46:46.269277 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.269256 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed-metrics-tls\") pod \"dns-default-4vnfl\" (UID: \"e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed\") " pod="openshift-dns/dns-default-4vnfl" Jul 17 21:46:46.273714 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.273544 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcmnm\" (UniqueName: \"kubernetes.io/projected/e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed-kube-api-access-rcmnm\") pod \"dns-default-4vnfl\" (UID: \"e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed\") " pod="openshift-dns/dns-default-4vnfl" Jul 17 21:46:46.274122 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.274099 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r9t5\" (UniqueName: \"kubernetes.io/projected/aac16a59-a485-4da6-8093-78fb71a06bcd-kube-api-access-7r9t5\") pod \"insights-runtime-extractor-ntxm7\" (UID: \"aac16a59-a485-4da6-8093-78fb71a06bcd\") " pod="openshift-insights/insights-runtime-extractor-ntxm7" Jul 17 21:46:46.279922 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.279905 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/aac16a59-a485-4da6-8093-78fb71a06bcd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ntxm7\" (UID: \"aac16a59-a485-4da6-8093-78fb71a06bcd\") " pod="openshift-insights/insights-runtime-extractor-ntxm7" Jul 17 21:46:46.337062 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.337041 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ntxm7" Jul 17 21:46:46.350196 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.350176 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4vnfl" Jul 17 21:46:46.366109 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.366087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm9cz\" (UniqueName: \"kubernetes.io/projected/67344770-d2f2-44bb-a026-130469fe7114-kube-api-access-xm9cz\") pod \"ingress-canary-4pwvp\" (UID: \"67344770-d2f2-44bb-a026-130469fe7114\") " pod="openshift-ingress-canary/ingress-canary-4pwvp" Jul 17 21:46:46.366186 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.366128 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67344770-d2f2-44bb-a026-130469fe7114-cert\") pod \"ingress-canary-4pwvp\" (UID: \"67344770-d2f2-44bb-a026-130469fe7114\") " pod="openshift-ingress-canary/ingress-canary-4pwvp" Jul 17 21:46:46.368480 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.368461 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67344770-d2f2-44bb-a026-130469fe7114-cert\") pod \"ingress-canary-4pwvp\" (UID: \"67344770-d2f2-44bb-a026-130469fe7114\") " pod="openshift-ingress-canary/ingress-canary-4pwvp" Jul 17 21:46:46.375910 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.375894 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm9cz\" (UniqueName: \"kubernetes.io/projected/67344770-d2f2-44bb-a026-130469fe7114-kube-api-access-xm9cz\") pod \"ingress-canary-4pwvp\" (UID: \"67344770-d2f2-44bb-a026-130469fe7114\") " pod="openshift-ingress-canary/ingress-canary-4pwvp" Jul 17 21:46:46.442094 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.442063 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4pwvp" Jul 17 21:46:46.545132 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.545083 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ntxm7"] Jul 17 21:46:46.549157 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.549131 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4vnfl"] Jul 17 21:46:46.549629 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:46.549604 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac16a59_a485_4da6_8093_78fb71a06bcd.slice/crio-e023068420ee444ca81513a2a8e465d673c6f575a30f7dc04681b57ac4358860 WatchSource:0}: Error finding container e023068420ee444ca81513a2a8e465d673c6f575a30f7dc04681b57ac4358860: Status 404 returned error can't find the container with id e023068420ee444ca81513a2a8e465d673c6f575a30f7dc04681b57ac4358860 Jul 17 21:46:46.604898 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.604822 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4pwvp"] Jul 17 21:46:46.608805 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:46.608778 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67344770_d2f2_44bb_a026_130469fe7114.slice/crio-cf16c94dd2c2f47730e65ad6cbb98fbdea4cb9e7f998ae79386ca3a1974841b6 WatchSource:0}: Error finding container cf16c94dd2c2f47730e65ad6cbb98fbdea4cb9e7f998ae79386ca3a1974841b6: Status 404 returned error can't find the container with id cf16c94dd2c2f47730e65ad6cbb98fbdea4cb9e7f998ae79386ca3a1974841b6 Jul 17 21:46:46.886545 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.886443 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4pwvp" event={"ID":"67344770-d2f2-44bb-a026-130469fe7114","Type":"ContainerStarted","Data":"cf16c94dd2c2f47730e65ad6cbb98fbdea4cb9e7f998ae79386ca3a1974841b6"} Jul 17 21:46:46.887817 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.887737 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4vnfl" event={"ID":"e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed","Type":"ContainerStarted","Data":"bd7fc29be812f99a8d5837918b155257656e738105a0d8687d35e03692894cc3"} Jul 17 21:46:46.888912 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:46.888891 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ntxm7" event={"ID":"aac16a59-a485-4da6-8093-78fb71a06bcd","Type":"ContainerStarted","Data":"e023068420ee444ca81513a2a8e465d673c6f575a30f7dc04681b57ac4358860"} Jul 17 21:46:47.556664 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.556162 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-676c555f45-smpsd"] Jul 17 21:46:47.578630 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.578420 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-676c555f45-smpsd"] Jul 17 21:46:47.578630 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.578563 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.582599 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.582299 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 17 21:46:47.582599 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.582353 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hlhgr\"" Jul 17 21:46:47.582599 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.582299 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 17 21:46:47.583737 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.583715 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 17 21:46:47.584182 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.583980 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 17 21:46:47.584182 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.584096 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 17 21:46:47.584706 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.584354 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 17 21:46:47.584706 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.584577 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 17 21:46:47.591156 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.591139 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 17 21:46:47.676907 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.676869 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-console-oauth-config\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.677066 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.676963 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-trusted-ca-bundle\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.677066 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.676994 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-oauth-serving-cert\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.677066 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.677030 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-service-ca\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.677243 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.677069 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-console-serving-cert\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.677243 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.677092 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-console-config\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.677243 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.677133 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhxmh\" (UniqueName: \"kubernetes.io/projected/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-kube-api-access-bhxmh\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.699584 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.699558 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:46:47.699975 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.699960 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:46:47.700390 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.700367 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:46:47.702638 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.702617 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 21:46:47.702864 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.702846 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 21:46:47.703383 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.703366 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 21:46:47.703655 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.703638 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-28dc9\"" Jul 17 21:46:47.703726 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.703642 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 21:46:47.703726 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.703700 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mbv8k\"" Jul 17 21:46:47.780047 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.777503 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhxmh\" (UniqueName: \"kubernetes.io/projected/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-kube-api-access-bhxmh\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.780047 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.777554 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-console-oauth-config\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.780047 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.777615 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-trusted-ca-bundle\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.780047 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.777638 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-oauth-serving-cert\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.780047 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.777669 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-service-ca\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.780047 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.777693 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-console-serving-cert\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.780047 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.777710 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-console-config\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.780047 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.778276 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-console-config\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.780047 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.779265 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-oauth-serving-cert\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.780614 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.780429 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-trusted-ca-bundle\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.781113 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.781024 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-service-ca\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.783774 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.783743 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-console-oauth-config\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.786878 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.786856 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhxmh\" (UniqueName: \"kubernetes.io/projected/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-kube-api-access-bhxmh\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.791580 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.791559 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-console-serving-cert\") pod \"console-676c555f45-smpsd\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:47.894061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:47.893755 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:48.102805 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:48.102773 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-676c555f45-smpsd"] Jul 17 21:46:48.107998 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:48.107612 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b5eeaf8_bfdc_4e12_98ab_7e0ab8fcf728.slice/crio-ec0f50e7d9e02942f05c3aaf168360764163cd38863974d2acb7d5731afc88a8 WatchSource:0}: Error finding container ec0f50e7d9e02942f05c3aaf168360764163cd38863974d2acb7d5731afc88a8: Status 404 returned error can't find the container with id ec0f50e7d9e02942f05c3aaf168360764163cd38863974d2acb7d5731afc88a8 Jul 17 21:46:48.895218 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:48.894509 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r9qbw"] Jul 17 21:46:48.917986 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:48.917953 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676c555f45-smpsd" event={"ID":"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728","Type":"ContainerStarted","Data":"ec0f50e7d9e02942f05c3aaf168360764163cd38863974d2acb7d5731afc88a8"} Jul 17 21:46:48.918127 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:48.918000 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r9qbw"] Jul 17 21:46:48.918127 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:48.918091 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r9qbw" Jul 17 21:46:48.920608 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:48.920572 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 17 21:46:48.920833 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:48.920818 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7gjhv\"" Jul 17 21:46:48.990066 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:48.989932 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/783b4ba1-13ca-436f-bf60-81a4f5ff0e24-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-r9qbw\" (UID: \"783b4ba1-13ca-436f-bf60-81a4f5ff0e24\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r9qbw" Jul 17 21:46:49.090385 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:49.090345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/783b4ba1-13ca-436f-bf60-81a4f5ff0e24-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-r9qbw\" (UID: \"783b4ba1-13ca-436f-bf60-81a4f5ff0e24\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r9qbw" Jul 17 21:46:49.090659 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:49.090540 2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 17 21:46:49.090659 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:49.090624 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/783b4ba1-13ca-436f-bf60-81a4f5ff0e24-tls-certificates podName:783b4ba1-13ca-436f-bf60-81a4f5ff0e24 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:49.590600871 +0000 UTC m=+46.450824353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/783b4ba1-13ca-436f-bf60-81a4f5ff0e24-tls-certificates") pod "prometheus-operator-admission-webhook-85574b5f49-r9qbw" (UID: "783b4ba1-13ca-436f-bf60-81a4f5ff0e24") : secret "prometheus-operator-admission-webhook-tls" not found Jul 17 21:46:49.594301 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:49.594257 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/783b4ba1-13ca-436f-bf60-81a4f5ff0e24-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-r9qbw\" (UID: \"783b4ba1-13ca-436f-bf60-81a4f5ff0e24\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r9qbw" Jul 17 21:46:49.598598 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:49.598570 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/783b4ba1-13ca-436f-bf60-81a4f5ff0e24-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-r9qbw\" (UID: \"783b4ba1-13ca-436f-bf60-81a4f5ff0e24\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r9qbw" Jul 17 21:46:49.831622 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:49.831587 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r9qbw" Jul 17 21:46:55.114802 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:55.114635 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r9qbw"] Jul 17 21:46:55.118416 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:46:55.118394 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783b4ba1_13ca_436f_bf60_81a4f5ff0e24.slice/crio-8051aefc58cf07c97d0a06cef0f6a01a60eb56fd47d9f2a15a5caf95f2dd9a73 WatchSource:0}: Error finding container 8051aefc58cf07c97d0a06cef0f6a01a60eb56fd47d9f2a15a5caf95f2dd9a73: Status 404 returned error can't find the container with id 8051aefc58cf07c97d0a06cef0f6a01a60eb56fd47d9f2a15a5caf95f2dd9a73 Jul 17 21:46:55.919163 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:55.918743 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4pwvp" event={"ID":"67344770-d2f2-44bb-a026-130469fe7114","Type":"ContainerStarted","Data":"761cd8d10aa7fdff6e8cfb9506d6867584a0734fff123e5593ddcf51d9f7f1fb"} Jul 17 21:46:55.920794 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:55.920728 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ntxm7" event={"ID":"aac16a59-a485-4da6-8093-78fb71a06bcd","Type":"ContainerStarted","Data":"fd9a5372c33f047d33ecdfc681ee8d282528b96cbcc0bf0597853fa3d15ab01a"} Jul 17 21:46:55.922147 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:55.922117 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r9qbw" event={"ID":"783b4ba1-13ca-436f-bf60-81a4f5ff0e24","Type":"ContainerStarted","Data":"8051aefc58cf07c97d0a06cef0f6a01a60eb56fd47d9f2a15a5caf95f2dd9a73"} Jul 17 21:46:55.924734 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:55.924705 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676c555f45-smpsd" event={"ID":"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728","Type":"ContainerStarted","Data":"e1280e956af2c561d5fcefa0319ed1bca65c9cb10804eedc1c593c203ad0d2dc"} Jul 17 21:46:55.926681 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:55.926656 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4vnfl" event={"ID":"e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed","Type":"ContainerStarted","Data":"593826c0b3adc3a2f65b592f8208e2fae5807d32ea94ff2854d27aa66c8d1f73"} Jul 17 21:46:55.926792 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:55.926704 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4vnfl" event={"ID":"e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed","Type":"ContainerStarted","Data":"f78a36c14e0c3ac6b165f46d32fdf3504554e40f9447add0e1dd374c98e62040"} Jul 17 21:46:55.926847 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:55.926833 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4vnfl" Jul 17 21:46:55.934618 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:55.934581 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4pwvp" podStartSLOduration=1.589981021 podStartE2EDuration="9.934567783s" totalImagesPullingTime="8.344586762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:55.933407711 +0000 UTC m=+52.793631217" watchObservedRunningTime="2026-07-17 21:46:55.934567783 +0000 UTC m=+52.794791284" Jul 17 21:46:55.952465 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:55.952410 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-676c555f45-smpsd" podStartSLOduration=1.772030657 podStartE2EDuration="8.95239532s" totalImagesPullingTime="7.180364663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:46:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:55.952002705 +0000 UTC m=+52.812226206" watchObservedRunningTime="2026-07-17 21:46:55.95239532 +0000 UTC m=+52.812618803" Jul 17 21:46:57.894289 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:57.894105 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:57.894740 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:57.894331 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:57.900130 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:57.900109 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:57.921302 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:57.921242 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4vnfl" podStartSLOduration=4.525767903 podStartE2EDuration="12.921226192s" totalImagesPullingTime="8.395458289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:55.979468491 +0000 UTC m=+52.839692005" watchObservedRunningTime="2026-07-17 21:46:57.921226192 +0000 UTC m=+54.781449672" Jul 17 21:46:57.939594 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:57.939553 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r9qbw" event={"ID":"783b4ba1-13ca-436f-bf60-81a4f5ff0e24","Type":"ContainerStarted","Data":"7626fff5a1532cee2ed8d3cba1575fe6791f091ae11bfc0d87bfaed224b9153a"} Jul 17 21:46:57.939906 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:57.939885 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r9qbw" Jul 17 21:46:57.944447 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:57.944423 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:46:57.947186 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:57.947167 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r9qbw" Jul 17 21:46:57.957767 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:57.957712 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-r9qbw" podStartSLOduration=8.247044295 podStartE2EDuration="9.957693594s" totalImagesPullingTime="1.710649299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:57.955436335 +0000 UTC m=+54.815659838" watchObservedRunningTime="2026-07-17 21:46:57.957693594 +0000 UTC m=+54.817917100" Jul 17 21:46:59.030801 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.030769 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7"] Jul 17 21:46:59.102074 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.102039 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7"] Jul 17 21:46:59.102222 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.102182 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7" Jul 17 21:46:59.104947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.104922 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 17 21:46:59.105077 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.105058 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 17 21:46:59.105792 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.105769 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-trt5x\"" Jul 17 21:46:59.105935 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.105806 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 17 21:46:59.106061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.106042 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 17 21:46:59.106181 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.106136 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 17 21:46:59.164571 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.164537 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdd2w\" (UniqueName: \"kubernetes.io/projected/b2be724c-9805-43fc-a2b4-a131ceefae54-kube-api-access-zdd2w\") pod \"prometheus-operator-5945b5cfd-8wkc7\" (UID: \"b2be724c-9805-43fc-a2b4-a131ceefae54\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7" Jul 17 21:46:59.164765 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.164589 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2be724c-9805-43fc-a2b4-a131ceefae54-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-8wkc7\" (UID: \"b2be724c-9805-43fc-a2b4-a131ceefae54\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7" Jul 17 21:46:59.164765 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.164659 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2be724c-9805-43fc-a2b4-a131ceefae54-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-8wkc7\" (UID: \"b2be724c-9805-43fc-a2b4-a131ceefae54\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7" Jul 17 21:46:59.164765 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.164697 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2be724c-9805-43fc-a2b4-a131ceefae54-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8wkc7\" (UID: \"b2be724c-9805-43fc-a2b4-a131ceefae54\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7" Jul 17 21:46:59.265439 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.265403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2be724c-9805-43fc-a2b4-a131ceefae54-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-8wkc7\" (UID: \"b2be724c-9805-43fc-a2b4-a131ceefae54\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7" Jul 17 21:46:59.265625 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.265453 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2be724c-9805-43fc-a2b4-a131ceefae54-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-8wkc7\" (UID: \"b2be724c-9805-43fc-a2b4-a131ceefae54\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7" Jul 17 21:46:59.265625 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.265485 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2be724c-9805-43fc-a2b4-a131ceefae54-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8wkc7\" (UID: \"b2be724c-9805-43fc-a2b4-a131ceefae54\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7" Jul 17 21:46:59.265733 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.265658 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdd2w\" (UniqueName: \"kubernetes.io/projected/b2be724c-9805-43fc-a2b4-a131ceefae54-kube-api-access-zdd2w\") pod \"prometheus-operator-5945b5cfd-8wkc7\" (UID: \"b2be724c-9805-43fc-a2b4-a131ceefae54\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7" Jul 17 21:46:59.265733 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:59.265577 2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 17 21:46:59.265829 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:46:59.265745 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2be724c-9805-43fc-a2b4-a131ceefae54-prometheus-operator-tls podName:b2be724c-9805-43fc-a2b4-a131ceefae54 nodeName:}" failed. No retries permitted until 2026-07-17 21:46:59.765730357 +0000 UTC m=+56.625953837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/b2be724c-9805-43fc-a2b4-a131ceefae54-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-8wkc7" (UID: "b2be724c-9805-43fc-a2b4-a131ceefae54") : secret "prometheus-operator-tls" not found Jul 17 21:46:59.266232 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.266206 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2be724c-9805-43fc-a2b4-a131ceefae54-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-8wkc7\" (UID: \"b2be724c-9805-43fc-a2b4-a131ceefae54\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7" Jul 17 21:46:59.269921 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.269901 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2be724c-9805-43fc-a2b4-a131ceefae54-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-8wkc7\" (UID: \"b2be724c-9805-43fc-a2b4-a131ceefae54\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7" Jul 17 21:46:59.276823 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.276801 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdd2w\" (UniqueName: \"kubernetes.io/projected/b2be724c-9805-43fc-a2b4-a131ceefae54-kube-api-access-zdd2w\") pod \"prometheus-operator-5945b5cfd-8wkc7\" (UID: \"b2be724c-9805-43fc-a2b4-a131ceefae54\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7" Jul 17 21:46:59.770096 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.770046 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2be724c-9805-43fc-a2b4-a131ceefae54-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8wkc7\" (UID: \"b2be724c-9805-43fc-a2b4-a131ceefae54\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7" Jul 17 21:46:59.772637 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.772618 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2be724c-9805-43fc-a2b4-a131ceefae54-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-8wkc7\" (UID: \"b2be724c-9805-43fc-a2b4-a131ceefae54\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7" Jul 17 21:46:59.947781 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:46:59.947747 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ntxm7" event={"ID":"aac16a59-a485-4da6-8093-78fb71a06bcd","Type":"ContainerStarted","Data":"1e75a19f98836112b77446dd5a0f985268081d541850207c2f830f72ca62beee"} Jul 17 21:47:00.012843 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:00.012806 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7" Jul 17 21:47:00.157308 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:00.157278 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7"] Jul 17 21:47:00.160443 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:47:00.160421 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2be724c_9805_43fc_a2b4_a131ceefae54.slice/crio-e4d8482c3aa9b67e12203dcb878376629aade3b2ab3e5faf8b344b3f5f1355ce WatchSource:0}: Error finding container e4d8482c3aa9b67e12203dcb878376629aade3b2ab3e5faf8b344b3f5f1355ce: Status 404 returned error can't find the container with id e4d8482c3aa9b67e12203dcb878376629aade3b2ab3e5faf8b344b3f5f1355ce Jul 17 21:47:00.952666 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:00.952620 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7" event={"ID":"b2be724c-9805-43fc-a2b4-a131ceefae54","Type":"ContainerStarted","Data":"e4d8482c3aa9b67e12203dcb878376629aade3b2ab3e5faf8b344b3f5f1355ce"} Jul 17 21:47:03.854938 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:03.854910 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-46wvc" Jul 17 21:47:03.966658 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:03.966619 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7" event={"ID":"b2be724c-9805-43fc-a2b4-a131ceefae54","Type":"ContainerStarted","Data":"e3e6b795fabe9e0f3283624908f5158e0d697f0ad245d51f07338089d83ed65c"} Jul 17 21:47:03.969384 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:03.969355 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ntxm7" event={"ID":"aac16a59-a485-4da6-8093-78fb71a06bcd","Type":"ContainerStarted","Data":"8078b70106ef9e6d96ec5d37c8893cd5c48d01d8429ddf4f409374f6b1b225b0"} Jul 17 21:47:03.987307 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:03.987251 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ntxm7" podStartSLOduration=1.942407019 podStartE2EDuration="18.987227457s" totalImagesPullingTime="17.044820438s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:46:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:03.986668981 +0000 UTC m=+60.846892522" watchObservedRunningTime="2026-07-17 21:47:03.987227457 +0000 UTC m=+60.847450958" Jul 17 21:47:04.974027 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:04.973992 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7" event={"ID":"b2be724c-9805-43fc-a2b4-a131ceefae54","Type":"ContainerStarted","Data":"1c34027c1846dad1f49936fd650f3b99a87fae8000da9cecda18257b89f3e2c0"} Jul 17 21:47:04.991410 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:04.991368 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-8wkc7" podStartSLOduration=2.275131155 podStartE2EDuration="5.991354007s" totalImagesPullingTime="3.716222852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:04.990911618 +0000 UTC m=+61.851135132" watchObservedRunningTime="2026-07-17 21:47:04.991354007 +0000 UTC m=+61.851577509" Jul 17 21:47:05.931608 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:05.931565 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4vnfl" Jul 17 21:47:07.332472 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.332439 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8"] Jul 17 21:47:07.334647 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.334631 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8" Jul 17 21:47:07.336690 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.336669 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 17 21:47:07.336825 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.336764 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gjj97\"" Jul 17 21:47:07.336825 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.336763 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 17 21:47:07.343819 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.343796 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8"] Jul 17 21:47:07.355469 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.355437 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-crzrx"] Jul 17 21:47:07.358135 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.358114 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.360632 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.360599 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 17 21:47:07.360749 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.360642 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 17 21:47:07.360891 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.360876 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 17 21:47:07.360973 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.360901 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ng2r2\"" Jul 17 21:47:07.369257 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.369232 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-crzrx"] Jul 17 21:47:07.370097 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.370076 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-24f89"] Jul 17 21:47:07.372298 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.372279 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.374371 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.374354 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 17 21:47:07.374690 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.374672 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 17 21:47:07.374849 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.374820 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5ndnc\"" Jul 17 21:47:07.374947 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.374894 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 17 21:47:07.426037 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.426008 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-sys\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.426037 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.426042 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/402b046d-053a-4887-a6b9-ec2dbf686caf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-crzrx\" (UID: \"402b046d-053a-4887-a6b9-ec2dbf686caf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.426252 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.426071 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-root\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.426252 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.426099 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-node-exporter-wtmp\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.426252 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.426135 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/402b046d-053a-4887-a6b9-ec2dbf686caf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-crzrx\" (UID: \"402b046d-053a-4887-a6b9-ec2dbf686caf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.426252 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.426187 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1d9d9a0-5aca-4690-9d56-519a5e3791bb-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-sw4s8\" (UID: \"c1d9d9a0-5aca-4690-9d56-519a5e3791bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8" Jul 17 21:47:07.426252 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.426242 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/402b046d-053a-4887-a6b9-ec2dbf686caf-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-crzrx\" (UID: \"402b046d-053a-4887-a6b9-ec2dbf686caf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.426429 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.426274 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-metrics-client-ca\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.426429 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.426290 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.426429 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.426306 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9drh\" (UniqueName: \"kubernetes.io/projected/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-kube-api-access-s9drh\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.426429 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.426349 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1d9d9a0-5aca-4690-9d56-519a5e3791bb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-sw4s8\" (UID: \"c1d9d9a0-5aca-4690-9d56-519a5e3791bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8" Jul 17 21:47:07.426429 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.426368 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-node-exporter-accelerators-collector-config\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.426429 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.426388 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-node-exporter-textfile\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.426429 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.426412 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx2kh\" (UniqueName: \"kubernetes.io/projected/c1d9d9a0-5aca-4690-9d56-519a5e3791bb-kube-api-access-xx2kh\") pod \"openshift-state-metrics-79f6958cdf-sw4s8\" (UID: \"c1d9d9a0-5aca-4690-9d56-519a5e3791bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8" Jul 17 21:47:07.426429 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.426429 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/402b046d-053a-4887-a6b9-ec2dbf686caf-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-crzrx\" (UID: \"402b046d-053a-4887-a6b9-ec2dbf686caf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.426740 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.426448 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/402b046d-053a-4887-a6b9-ec2dbf686caf-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-crzrx\" (UID: \"402b046d-053a-4887-a6b9-ec2dbf686caf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.426740 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.426464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-node-exporter-tls\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.426740 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.426518 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1d9d9a0-5aca-4690-9d56-519a5e3791bb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-sw4s8\" (UID: \"c1d9d9a0-5aca-4690-9d56-519a5e3791bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8" Jul 17 21:47:07.426740 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.426541 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld2n5\" (UniqueName: \"kubernetes.io/projected/402b046d-053a-4887-a6b9-ec2dbf686caf-kube-api-access-ld2n5\") pod \"kube-state-metrics-56c5594674-crzrx\" (UID: \"402b046d-053a-4887-a6b9-ec2dbf686caf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.527449 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.527412 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-root\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.527660 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.527454 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-node-exporter-wtmp\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.527660 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.527512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/402b046d-053a-4887-a6b9-ec2dbf686caf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-crzrx\" (UID: \"402b046d-053a-4887-a6b9-ec2dbf686caf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.527660 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.527534 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-root\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.527660 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.527562 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1d9d9a0-5aca-4690-9d56-519a5e3791bb-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-sw4s8\" (UID: \"c1d9d9a0-5aca-4690-9d56-519a5e3791bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8" Jul 17 21:47:07.527660 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.527602 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/402b046d-053a-4887-a6b9-ec2dbf686caf-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-crzrx\" (UID: \"402b046d-053a-4887-a6b9-ec2dbf686caf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.527660 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.527628 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-metrics-client-ca\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.527660 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.527650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.528012 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.527664 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-node-exporter-wtmp\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.528012 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.527678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9drh\" (UniqueName: \"kubernetes.io/projected/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-kube-api-access-s9drh\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.528012 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.527741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1d9d9a0-5aca-4690-9d56-519a5e3791bb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-sw4s8\" (UID: \"c1d9d9a0-5aca-4690-9d56-519a5e3791bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8" Jul 17 21:47:07.528168 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.528045 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/402b046d-053a-4887-a6b9-ec2dbf686caf-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-crzrx\" (UID: \"402b046d-053a-4887-a6b9-ec2dbf686caf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.528168 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.527771 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-node-exporter-accelerators-collector-config\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.528280 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.528186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-node-exporter-textfile\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.528337 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.528288 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-metrics-client-ca\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.528403 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.528344 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx2kh\" (UniqueName: \"kubernetes.io/projected/c1d9d9a0-5aca-4690-9d56-519a5e3791bb-kube-api-access-xx2kh\") pod \"openshift-state-metrics-79f6958cdf-sw4s8\" (UID: \"c1d9d9a0-5aca-4690-9d56-519a5e3791bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8" Jul 17 21:47:07.528464 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.528400 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/402b046d-053a-4887-a6b9-ec2dbf686caf-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-crzrx\" (UID: \"402b046d-053a-4887-a6b9-ec2dbf686caf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.528464 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.528432 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/402b046d-053a-4887-a6b9-ec2dbf686caf-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-crzrx\" (UID: \"402b046d-053a-4887-a6b9-ec2dbf686caf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.528464 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.528457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-node-exporter-tls\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.528652 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.528457 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-node-exporter-textfile\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.528652 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.528426 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c1d9d9a0-5aca-4690-9d56-519a5e3791bb-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-sw4s8\" (UID: \"c1d9d9a0-5aca-4690-9d56-519a5e3791bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8" Jul 17 21:47:07.528652 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.528585 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-node-exporter-accelerators-collector-config\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.528801 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:47:07.528681 2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 17 21:47:07.528801 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:47:07.528750 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-node-exporter-tls podName:63b9d2a6-f144-44b6-ad72-8f2bdacf512c nodeName:}" failed. No retries permitted until 2026-07-17 21:47:08.028730102 +0000 UTC m=+64.888953584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-node-exporter-tls") pod "node-exporter-24f89" (UID: "63b9d2a6-f144-44b6-ad72-8f2bdacf512c") : secret "node-exporter-tls" not found Jul 17 21:47:07.528914 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.528825 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1d9d9a0-5aca-4690-9d56-519a5e3791bb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-sw4s8\" (UID: \"c1d9d9a0-5aca-4690-9d56-519a5e3791bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8" Jul 17 21:47:07.528914 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.528863 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld2n5\" (UniqueName: \"kubernetes.io/projected/402b046d-053a-4887-a6b9-ec2dbf686caf-kube-api-access-ld2n5\") pod \"kube-state-metrics-56c5594674-crzrx\" (UID: \"402b046d-053a-4887-a6b9-ec2dbf686caf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.528914 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.528898 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-sys\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.529077 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.528930 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/402b046d-053a-4887-a6b9-ec2dbf686caf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-crzrx\" (UID: \"402b046d-053a-4887-a6b9-ec2dbf686caf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.529077 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.529063 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/402b046d-053a-4887-a6b9-ec2dbf686caf-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-crzrx\" (UID: \"402b046d-053a-4887-a6b9-ec2dbf686caf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.529195 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.529118 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-sys\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.529797 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.529773 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/402b046d-053a-4887-a6b9-ec2dbf686caf-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-crzrx\" (UID: \"402b046d-053a-4887-a6b9-ec2dbf686caf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.530518 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.530467 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.530824 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.530803 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c1d9d9a0-5aca-4690-9d56-519a5e3791bb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-sw4s8\" (UID: \"c1d9d9a0-5aca-4690-9d56-519a5e3791bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8" Jul 17 21:47:07.530913 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.530832 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/402b046d-053a-4887-a6b9-ec2dbf686caf-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-crzrx\" (UID: \"402b046d-053a-4887-a6b9-ec2dbf686caf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.531307 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.531280 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/402b046d-053a-4887-a6b9-ec2dbf686caf-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-crzrx\" (UID: \"402b046d-053a-4887-a6b9-ec2dbf686caf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.531421 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.531403 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c1d9d9a0-5aca-4690-9d56-519a5e3791bb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-sw4s8\" (UID: \"c1d9d9a0-5aca-4690-9d56-519a5e3791bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8" Jul 17 21:47:07.535647 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.535623 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9drh\" (UniqueName: \"kubernetes.io/projected/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-kube-api-access-s9drh\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:07.536899 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.536871 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx2kh\" (UniqueName: \"kubernetes.io/projected/c1d9d9a0-5aca-4690-9d56-519a5e3791bb-kube-api-access-xx2kh\") pod \"openshift-state-metrics-79f6958cdf-sw4s8\" (UID: \"c1d9d9a0-5aca-4690-9d56-519a5e3791bb\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8" Jul 17 21:47:07.537131 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.537116 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld2n5\" (UniqueName: \"kubernetes.io/projected/402b046d-053a-4887-a6b9-ec2dbf686caf-kube-api-access-ld2n5\") pod \"kube-state-metrics-56c5594674-crzrx\" (UID: \"402b046d-053a-4887-a6b9-ec2dbf686caf\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.643884 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.643817 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8" Jul 17 21:47:07.666997 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.666894 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" Jul 17 21:47:07.788200 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.788178 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8"] Jul 17 21:47:07.790121 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:47:07.790097 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d9d9a0_5aca_4690_9d56_519a5e3791bb.slice/crio-c5dc3fa2e524a20ba1f27e0109c4ac6d4112a4a6980c2dede06a91a062e2880b WatchSource:0}: Error finding container c5dc3fa2e524a20ba1f27e0109c4ac6d4112a4a6980c2dede06a91a062e2880b: Status 404 returned error can't find the container with id c5dc3fa2e524a20ba1f27e0109c4ac6d4112a4a6980c2dede06a91a062e2880b Jul 17 21:47:07.804331 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.804310 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-crzrx"] Jul 17 21:47:07.807776 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:47:07.807755 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402b046d_053a_4887_a6b9_ec2dbf686caf.slice/crio-270cd5df23a2967ae45f031852c6b0824b2e0a4ac354d9a8565a1d730c8eb0b9 WatchSource:0}: Error finding container 270cd5df23a2967ae45f031852c6b0824b2e0a4ac354d9a8565a1d730c8eb0b9: Status 404 returned error can't find the container with id 270cd5df23a2967ae45f031852c6b0824b2e0a4ac354d9a8565a1d730c8eb0b9 Jul 17 21:47:07.984953 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.984921 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" event={"ID":"402b046d-053a-4887-a6b9-ec2dbf686caf","Type":"ContainerStarted","Data":"270cd5df23a2967ae45f031852c6b0824b2e0a4ac354d9a8565a1d730c8eb0b9"} Jul 17 21:47:07.986475 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.986442 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8" event={"ID":"c1d9d9a0-5aca-4690-9d56-519a5e3791bb","Type":"ContainerStarted","Data":"2ddf73f224f804f094cb712fd0766beaa950b1e50e0da7b0ae128a29738f5ebd"} Jul 17 21:47:07.986607 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.986509 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8" event={"ID":"c1d9d9a0-5aca-4690-9d56-519a5e3791bb","Type":"ContainerStarted","Data":"ff9ac0769fda19e1318d4126168a676600be8af3f0d194a93ff7ba046ca1aad8"} Jul 17 21:47:07.986607 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:07.986532 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8" event={"ID":"c1d9d9a0-5aca-4690-9d56-519a5e3791bb","Type":"ContainerStarted","Data":"c5dc3fa2e524a20ba1f27e0109c4ac6d4112a4a6980c2dede06a91a062e2880b"} Jul 17 21:47:08.033065 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.033037 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-node-exporter-tls\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:08.035371 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.035347 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/63b9d2a6-f144-44b6-ad72-8f2bdacf512c-node-exporter-tls\") pod \"node-exporter-24f89\" (UID: \"63b9d2a6-f144-44b6-ad72-8f2bdacf512c\") " pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:08.281341 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.281309 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-24f89" Jul 17 21:47:08.292932 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:47:08.292899 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b9d2a6_f144_44b6_ad72_8f2bdacf512c.slice/crio-f3ab822dcd717689023306ead78df3b2793ccfb0e38e411b6e3ef10746232821 WatchSource:0}: Error finding container f3ab822dcd717689023306ead78df3b2793ccfb0e38e411b6e3ef10746232821: Status 404 returned error can't find the container with id f3ab822dcd717689023306ead78df3b2793ccfb0e38e411b6e3ef10746232821 Jul 17 21:47:08.409479 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.409446 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 21:47:08.412542 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.412517 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.415382 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.415005 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 17 21:47:08.415382 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.415015 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 17 21:47:08.415382 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.415031 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 17 21:47:08.415382 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.415106 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 17 21:47:08.415382 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.415264 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 17 21:47:08.415382 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.415338 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xqr4l\"" Jul 17 21:47:08.415382 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.415354 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 17 21:47:08.415703 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.415554 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 17 21:47:08.415703 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.415590 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 17 21:47:08.415807 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.415708 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 17 21:47:08.431459 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.431423 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 21:47:08.437107 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.437083 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.437236 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.437127 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/284136da-9486-4e1f-a654-ac4459450bbf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.437236 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.437154 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-web-config\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.437236 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.437177 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/284136da-9486-4e1f-a654-ac4459450bbf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.437236 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.437202 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7vhr\" (UniqueName: \"kubernetes.io/projected/284136da-9486-4e1f-a654-ac4459450bbf-kube-api-access-p7vhr\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.437810 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.437786 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.437934 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.437823 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/284136da-9486-4e1f-a654-ac4459450bbf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.437934 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.437850 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-config-volume\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.437934 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.437873 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.437934 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.437919 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.438143 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.437966 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.438143 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.437992 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/284136da-9486-4e1f-a654-ac4459450bbf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.438143 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.438026 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/284136da-9486-4e1f-a654-ac4459450bbf-config-out\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.539821 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.538442 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/284136da-9486-4e1f-a654-ac4459450bbf-config-out\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.539821 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.538532 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.539821 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.538568 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/284136da-9486-4e1f-a654-ac4459450bbf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.539821 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.538587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-web-config\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.539821 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.538606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/284136da-9486-4e1f-a654-ac4459450bbf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.539821 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.538643 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7vhr\" (UniqueName: \"kubernetes.io/projected/284136da-9486-4e1f-a654-ac4459450bbf-kube-api-access-p7vhr\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.539821 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.538721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.539821 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.538749 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/284136da-9486-4e1f-a654-ac4459450bbf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.539821 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.538773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-config-volume\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.539821 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.538799 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.539821 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.538850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.539821 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.538892 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.539821 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.538909 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/284136da-9486-4e1f-a654-ac4459450bbf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.539821 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.539310 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/284136da-9486-4e1f-a654-ac4459450bbf-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.542559 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:47:08.540730 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/284136da-9486-4e1f-a654-ac4459450bbf-alertmanager-trusted-ca-bundle podName:284136da-9486-4e1f-a654-ac4459450bbf nodeName:}" failed. No retries permitted until 2026-07-17 21:47:09.040706786 +0000 UTC m=+65.900930283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/284136da-9486-4e1f-a654-ac4459450bbf-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "284136da-9486-4e1f-a654-ac4459450bbf") : configmap references non-existent config key: ca-bundle.crt Jul 17 21:47:08.542559 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:47:08.542309 2577 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 17 21:47:08.542559 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:47:08.542369 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-secret-alertmanager-main-tls podName:284136da-9486-4e1f-a654-ac4459450bbf nodeName:}" failed. No retries permitted until 2026-07-17 21:47:09.042351728 +0000 UTC m=+65.902575214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "284136da-9486-4e1f-a654-ac4459450bbf") : secret "alertmanager-main-tls" not found Jul 17 21:47:08.542800 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.542703 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/284136da-9486-4e1f-a654-ac4459450bbf-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.547519 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.547416 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/284136da-9486-4e1f-a654-ac4459450bbf-config-out\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.547973 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.547907 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-config-volume\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.547973 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.547926 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-web-config\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.549005 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.548208 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/284136da-9486-4e1f-a654-ac4459450bbf-tls-assets\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.549005 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.548906 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.549005 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.548906 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.550712 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.550673 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7vhr\" (UniqueName: \"kubernetes.io/projected/284136da-9486-4e1f-a654-ac4459450bbf-kube-api-access-p7vhr\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.551824 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.551801 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.551920 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.551853 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:08.992592 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:08.992552 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-24f89" event={"ID":"63b9d2a6-f144-44b6-ad72-8f2bdacf512c","Type":"ContainerStarted","Data":"f3ab822dcd717689023306ead78df3b2793ccfb0e38e411b6e3ef10746232821"} Jul 17 21:47:09.042882 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.042842 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/284136da-9486-4e1f-a654-ac4459450bbf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:09.043078 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.042942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:09.043780 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.043756 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/284136da-9486-4e1f-a654-ac4459450bbf-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:09.045960 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.045937 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/284136da-9486-4e1f-a654-ac4459450bbf-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"284136da-9486-4e1f-a654-ac4459450bbf\") " pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:09.325686 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.325654 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 17 21:47:09.331097 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.331071 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8666d66855-vl4vv"] Jul 17 21:47:09.334776 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.334752 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.339098 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.339075 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7r5kb\"" Jul 17 21:47:09.339182 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.339160 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 17 21:47:09.339325 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.339307 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 17 21:47:09.339794 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.339407 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 17 21:47:09.339794 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.339539 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 17 21:47:09.339794 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.339694 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-913ajg9flqjvc\"" Jul 17 21:47:09.340768 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.340235 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 17 21:47:09.345167 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.345146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs\") pod \"network-metrics-daemon-4vfnr\" (UID: \"491a8723-75cd-4309-abad-792b39758916\") " pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:47:09.350424 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.350401 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 17 21:47:09.358282 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.358226 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/491a8723-75cd-4309-abad-792b39758916-metrics-certs\") pod \"network-metrics-daemon-4vfnr\" (UID: \"491a8723-75cd-4309-abad-792b39758916\") " pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:47:09.361085 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.361054 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8666d66855-vl4vv"] Jul 17 21:47:09.446223 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.446193 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/512670a9-3155-4a10-85c9-d4d4826d32e9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.446593 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.446240 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qngkj\" (UniqueName: \"kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj\") pod \"network-check-target-kp5bb\" (UID: \"ee877ecd-672d-4415-a69e-831da8e098fe\") " pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:47:09.446593 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.446294 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxbfc\" (UniqueName: \"kubernetes.io/projected/512670a9-3155-4a10-85c9-d4d4826d32e9-kube-api-access-cxbfc\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.446593 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.446317 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/512670a9-3155-4a10-85c9-d4d4826d32e9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.446593 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.446369 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/512670a9-3155-4a10-85c9-d4d4826d32e9-metrics-client-ca\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.446593 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.446398 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/512670a9-3155-4a10-85c9-d4d4826d32e9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.446593 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.446435 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/512670a9-3155-4a10-85c9-d4d4826d32e9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.446593 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.446474 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret\") pod \"global-pull-secret-syncer-982gr\" (UID: \"e2430562-9210-4afd-9800-9e66f9548cb7\") " pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:47:09.446593 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.446514 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/512670a9-3155-4a10-85c9-d4d4826d32e9-secret-thanos-querier-tls\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.446593 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.446532 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/512670a9-3155-4a10-85c9-d4d4826d32e9-secret-grpc-tls\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.450194 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.450174 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 17 21:47:09.450440 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.450427 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 17 21:47:09.460391 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.459544 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e2430562-9210-4afd-9800-9e66f9548cb7-original-pull-secret\") pod \"global-pull-secret-syncer-982gr\" (UID: \"e2430562-9210-4afd-9800-9e66f9548cb7\") " pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:47:09.460391 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.460044 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 17 21:47:09.471285 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.471242 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qngkj\" (UniqueName: \"kubernetes.io/projected/ee877ecd-672d-4415-a69e-831da8e098fe-kube-api-access-qngkj\") pod \"network-check-target-kp5bb\" (UID: \"ee877ecd-672d-4415-a69e-831da8e098fe\") " pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:47:09.530872 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.530789 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 17 21:47:09.535661 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:47:09.535632 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284136da_9486_4e1f_a654_ac4459450bbf.slice/crio-fee9975b8390eeded4f6b25304dac663d006a368f8a110bcfd3766b279f75041 WatchSource:0}: Error finding container fee9975b8390eeded4f6b25304dac663d006a368f8a110bcfd3766b279f75041: Status 404 returned error can't find the container with id fee9975b8390eeded4f6b25304dac663d006a368f8a110bcfd3766b279f75041 Jul 17 21:47:09.547561 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.547529 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/512670a9-3155-4a10-85c9-d4d4826d32e9-metrics-client-ca\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.547666 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.547580 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/512670a9-3155-4a10-85c9-d4d4826d32e9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.547666 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.547632 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/512670a9-3155-4a10-85c9-d4d4826d32e9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.547774 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.547685 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/512670a9-3155-4a10-85c9-d4d4826d32e9-secret-thanos-querier-tls\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.548537 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.548154 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/512670a9-3155-4a10-85c9-d4d4826d32e9-secret-grpc-tls\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.548537 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.548210 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/512670a9-3155-4a10-85c9-d4d4826d32e9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.548537 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.548254 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxbfc\" (UniqueName: \"kubernetes.io/projected/512670a9-3155-4a10-85c9-d4d4826d32e9-kube-api-access-cxbfc\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.548537 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.548288 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/512670a9-3155-4a10-85c9-d4d4826d32e9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.548537 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.548289 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/512670a9-3155-4a10-85c9-d4d4826d32e9-metrics-client-ca\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.551693 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.551650 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/512670a9-3155-4a10-85c9-d4d4826d32e9-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.552097 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.552075 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/512670a9-3155-4a10-85c9-d4d4826d32e9-secret-thanos-querier-tls\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.552190 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.552160 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/512670a9-3155-4a10-85c9-d4d4826d32e9-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.552704 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.552661 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/512670a9-3155-4a10-85c9-d4d4826d32e9-secret-grpc-tls\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.553838 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.553799 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/512670a9-3155-4a10-85c9-d4d4826d32e9-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.555149 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.555104 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/512670a9-3155-4a10-85c9-d4d4826d32e9-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.556572 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.556551 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxbfc\" (UniqueName: \"kubernetes.io/projected/512670a9-3155-4a10-85c9-d4d4826d32e9-kube-api-access-cxbfc\") pod \"thanos-querier-8666d66855-vl4vv\" (UID: \"512670a9-3155-4a10-85c9-d4d4826d32e9\") " pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:09.615560 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.615528 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-982gr" Jul 17 21:47:09.625600 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.625577 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mbv8k\"" Jul 17 21:47:09.632703 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.632680 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-28dc9\"" Jul 17 21:47:09.633729 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.633713 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4vfnr" Jul 17 21:47:09.641538 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.641521 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:47:09.653440 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:09.650679 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:10.001813 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:10.001678 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" event={"ID":"402b046d-053a-4887-a6b9-ec2dbf686caf","Type":"ContainerStarted","Data":"6ff1159200ce124d7cc39b91c1d01e4164a43c27164e7bed94f590f8d4f9b99d"} Jul 17 21:47:10.001813 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:10.001747 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" event={"ID":"402b046d-053a-4887-a6b9-ec2dbf686caf","Type":"ContainerStarted","Data":"39f5c63db91fe8429832c72023469712ce338e8c4b32bbe6e30553d00fabed26"} Jul 17 21:47:10.001813 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:10.001777 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" event={"ID":"402b046d-053a-4887-a6b9-ec2dbf686caf","Type":"ContainerStarted","Data":"582c704c367ca6ae6691aa07077efef01d36c3eaf6a0a193d34d409e7ccb673d"} Jul 17 21:47:10.005787 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:10.005295 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8" event={"ID":"c1d9d9a0-5aca-4690-9d56-519a5e3791bb","Type":"ContainerStarted","Data":"9b0be63d1dc5bbf688f6c93f00bd5f5abbe558b4a1bfcdc4161fbe7ea16b6b17"} Jul 17 21:47:10.013204 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:10.012841 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8666d66855-vl4vv"] Jul 17 21:47:10.013737 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:47:10.013713 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod512670a9_3155_4a10_85c9_d4d4826d32e9.slice/crio-8945739bec68a838474f0c9f972262b50c93b296228553fd90101140c3a05ad7 WatchSource:0}: Error finding container 8945739bec68a838474f0c9f972262b50c93b296228553fd90101140c3a05ad7: Status 404 returned error can't find the container with id 8945739bec68a838474f0c9f972262b50c93b296228553fd90101140c3a05ad7 Jul 17 21:47:10.014917 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:10.014894 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"284136da-9486-4e1f-a654-ac4459450bbf","Type":"ContainerStarted","Data":"fee9975b8390eeded4f6b25304dac663d006a368f8a110bcfd3766b279f75041"} Jul 17 21:47:10.017670 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:10.017426 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-24f89" event={"ID":"63b9d2a6-f144-44b6-ad72-8f2bdacf512c","Type":"ContainerStarted","Data":"6d44e445d40bae02382ecf6ebd997646807ed3068e78069a7ff2602879a9be06"} Jul 17 21:47:10.026701 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:10.025565 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-crzrx" podStartSLOduration=1.6188856249999999 podStartE2EDuration="3.025549729s" totalImagesPullingTime="1.406664104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:10.025532035 +0000 UTC m=+66.885755561" watchObservedRunningTime="2026-07-17 21:47:10.025549729 +0000 UTC m=+66.885773230" Jul 17 21:47:10.047232 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:10.047178 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-sw4s8" podStartSLOduration=1.733725094 podStartE2EDuration="3.047163379s" totalImagesPullingTime="1.313438285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:10.045852479 +0000 UTC m=+66.906075982" watchObservedRunningTime="2026-07-17 21:47:10.047163379 +0000 UTC m=+66.907386880" Jul 17 21:47:10.065286 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:10.065032 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-982gr"] Jul 17 21:47:10.100533 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:47:10.100439 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2430562_9210_4afd_9800_9e66f9548cb7.slice/crio-f3918c6ca544508a6cb086e7329b491e3c10526d9a8e9250b98ed98a1709f5fc WatchSource:0}: Error finding container f3918c6ca544508a6cb086e7329b491e3c10526d9a8e9250b98ed98a1709f5fc: Status 404 returned error can't find the container with id f3918c6ca544508a6cb086e7329b491e3c10526d9a8e9250b98ed98a1709f5fc Jul 17 21:47:10.223953 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:10.223923 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kp5bb"] Jul 17 21:47:10.227376 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:10.227353 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4vfnr"] Jul 17 21:47:10.227777 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:47:10.227594 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee877ecd_672d_4415_a69e_831da8e098fe.slice/crio-5383672a5c6e7e19b7794a579fea8f4ca00ce503d198ea39b949074c1192f52f WatchSource:0}: Error finding container 5383672a5c6e7e19b7794a579fea8f4ca00ce503d198ea39b949074c1192f52f: Status 404 returned error can't find the container with id 5383672a5c6e7e19b7794a579fea8f4ca00ce503d198ea39b949074c1192f52f Jul 17 21:47:11.024699 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.024660 2577 generic.go:356] "Generic (PLEG): container finished" podID="63b9d2a6-f144-44b6-ad72-8f2bdacf512c" containerID="6d44e445d40bae02382ecf6ebd997646807ed3068e78069a7ff2602879a9be06" exitCode=0 Jul 17 21:47:11.025155 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.024747 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-24f89" event={"ID":"63b9d2a6-f144-44b6-ad72-8f2bdacf512c","Type":"ContainerDied","Data":"6d44e445d40bae02382ecf6ebd997646807ed3068e78069a7ff2602879a9be06"} Jul 17 21:47:11.028191 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.028163 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-982gr" event={"ID":"e2430562-9210-4afd-9800-9e66f9548cb7","Type":"ContainerStarted","Data":"f3918c6ca544508a6cb086e7329b491e3c10526d9a8e9250b98ed98a1709f5fc"} Jul 17 21:47:11.029986 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.029965 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4vfnr" event={"ID":"491a8723-75cd-4309-abad-792b39758916","Type":"ContainerStarted","Data":"849396c0ff7598ed9b62c84da232efca4571ffcc1285a1b8e5df7ef51d0e9e1e"} Jul 17 21:47:11.031774 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.031716 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kp5bb" event={"ID":"ee877ecd-672d-4415-a69e-831da8e098fe","Type":"ContainerStarted","Data":"5383672a5c6e7e19b7794a579fea8f4ca00ce503d198ea39b949074c1192f52f"} Jul 17 21:47:11.034585 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.034466 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" event={"ID":"512670a9-3155-4a10-85c9-d4d4826d32e9","Type":"ContainerStarted","Data":"8945739bec68a838474f0c9f972262b50c93b296228553fd90101140c3a05ad7"} Jul 17 21:47:11.039061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.038175 2577 generic.go:356] "Generic (PLEG): container finished" podID="284136da-9486-4e1f-a654-ac4459450bbf" containerID="468118c7cf920e9062a3ddf0e1abb5e08043abaaa79dd2333672b7547af7c765" exitCode=0 Jul 17 21:47:11.039061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.038363 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"284136da-9486-4e1f-a654-ac4459450bbf","Type":"ContainerDied","Data":"468118c7cf920e9062a3ddf0e1abb5e08043abaaa79dd2333672b7547af7c765"} Jul 17 21:47:11.752758 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.752472 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-85f79f4d9-w98td"] Jul 17 21:47:11.756415 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.755605 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.757847 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.757689 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 17 21:47:11.759653 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.758722 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 17 21:47:11.759653 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.758940 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 17 21:47:11.759653 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.759148 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 17 21:47:11.759653 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.759543 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-zswvn\"" Jul 17 21:47:11.759653 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.759562 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-d33ois44t3909\"" Jul 17 21:47:11.769822 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.768464 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85f79f4d9-w98td"] Jul 17 21:47:11.871363 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.871325 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/61876aa9-16d9-4016-b24c-f63fa89c32fb-metrics-server-audit-profiles\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.871548 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.871380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/61876aa9-16d9-4016-b24c-f63fa89c32fb-secret-metrics-server-tls\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.871548 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.871430 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/61876aa9-16d9-4016-b24c-f63fa89c32fb-secret-metrics-server-client-certs\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.871548 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.871455 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/61876aa9-16d9-4016-b24c-f63fa89c32fb-audit-log\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.871548 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.871478 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5pd9\" (UniqueName: \"kubernetes.io/projected/61876aa9-16d9-4016-b24c-f63fa89c32fb-kube-api-access-b5pd9\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.871770 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.871553 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61876aa9-16d9-4016-b24c-f63fa89c32fb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.871770 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.871585 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/61876aa9-16d9-4016-b24c-f63fa89c32fb-client-ca-bundle\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.972687 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.972651 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/61876aa9-16d9-4016-b24c-f63fa89c32fb-metrics-server-audit-profiles\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.972687 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.972689 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/61876aa9-16d9-4016-b24c-f63fa89c32fb-secret-metrics-server-tls\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.972869 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.972723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/61876aa9-16d9-4016-b24c-f63fa89c32fb-secret-metrics-server-client-certs\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.972869 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.972741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/61876aa9-16d9-4016-b24c-f63fa89c32fb-audit-log\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.972869 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.972757 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5pd9\" (UniqueName: \"kubernetes.io/projected/61876aa9-16d9-4016-b24c-f63fa89c32fb-kube-api-access-b5pd9\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.972869 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.972788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61876aa9-16d9-4016-b24c-f63fa89c32fb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.972869 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.972808 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/61876aa9-16d9-4016-b24c-f63fa89c32fb-client-ca-bundle\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.973558 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.973531 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/61876aa9-16d9-4016-b24c-f63fa89c32fb-audit-log\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.974137 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.974071 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/61876aa9-16d9-4016-b24c-f63fa89c32fb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.974588 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.974560 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/61876aa9-16d9-4016-b24c-f63fa89c32fb-metrics-server-audit-profiles\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.976341 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.976313 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/61876aa9-16d9-4016-b24c-f63fa89c32fb-client-ca-bundle\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.976847 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.976813 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/61876aa9-16d9-4016-b24c-f63fa89c32fb-secret-metrics-server-tls\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.977470 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.977425 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/61876aa9-16d9-4016-b24c-f63fa89c32fb-secret-metrics-server-client-certs\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:11.982325 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:11.982301 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5pd9\" (UniqueName: \"kubernetes.io/projected/61876aa9-16d9-4016-b24c-f63fa89c32fb-kube-api-access-b5pd9\") pod \"metrics-server-85f79f4d9-w98td\" (UID: \"61876aa9-16d9-4016-b24c-f63fa89c32fb\") " pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:12.047172 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.047137 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-24f89" event={"ID":"63b9d2a6-f144-44b6-ad72-8f2bdacf512c","Type":"ContainerStarted","Data":"1543fd2af016882ed5fb81d9c74a3d1a5b6b6b2a606c889c571dbf7c670a4102"} Jul 17 21:47:12.047658 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.047203 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-24f89" event={"ID":"63b9d2a6-f144-44b6-ad72-8f2bdacf512c","Type":"ContainerStarted","Data":"e0e1fde2c9db0fe2455032b86a20f3e0585fc34a18ed4a3db84b1b9ddc6111d3"} Jul 17 21:47:12.050376 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.050345 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4vfnr" event={"ID":"491a8723-75cd-4309-abad-792b39758916","Type":"ContainerStarted","Data":"de4503006083956ee6a10c2507f8fd081f60e19dee493ad826741799f3b326b7"} Jul 17 21:47:12.050528 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.050400 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4vfnr" event={"ID":"491a8723-75cd-4309-abad-792b39758916","Type":"ContainerStarted","Data":"ef0b8d8433ed26b50d04b7f5d6945d3a77f46495c1ca30d4a589f6ee247147d6"} Jul 17 21:47:12.071799 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.071746 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-24f89" podStartSLOduration=3.423758743 podStartE2EDuration="5.071726259s" totalImagesPullingTime="1.555599578s" totalInitContainerRuntime="92.367938ms" isStatefulPod=true podCreationTimestamp="2026-07-17 21:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:12.070111033 +0000 UTC m=+68.930334559" watchObservedRunningTime="2026-07-17 21:47:12.071726259 +0000 UTC m=+68.931949765" Jul 17 21:47:12.076683 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.076651 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:12.097637 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.097586 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4vfnr" podStartSLOduration=67.681751352 podStartE2EDuration="1m9.097567227s" totalImagesPullingTime="1.415815875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:46:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:12.094946822 +0000 UTC m=+68.955170324" watchObservedRunningTime="2026-07-17 21:47:12.097567227 +0000 UTC m=+68.957790728" Jul 17 21:47:12.151808 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.151773 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9ns75"] Jul 17 21:47:12.154781 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.154688 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9ns75" Jul 17 21:47:12.157775 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.157733 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 17 21:47:12.158277 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.158260 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-ddk7h\"" Jul 17 21:47:12.158639 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.158616 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cd949f4fd-vsfjk"] Jul 17 21:47:12.162177 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.161879 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.172610 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.172587 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9ns75"] Jul 17 21:47:12.175575 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.175523 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmtd4\" (UniqueName: \"kubernetes.io/projected/c3f8b396-45d6-403c-a699-9eb5d3d5c627-kube-api-access-tmtd4\") pod \"monitoring-plugin-7b7b7bc8bf-9ns75\" (UID: \"c3f8b396-45d6-403c-a699-9eb5d3d5c627\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9ns75" Jul 17 21:47:12.175681 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.175613 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3f8b396-45d6-403c-a699-9eb5d3d5c627-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-9ns75\" (UID: \"c3f8b396-45d6-403c-a699-9eb5d3d5c627\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9ns75" Jul 17 21:47:12.180230 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.180207 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cd949f4fd-vsfjk"] Jul 17 21:47:12.276584 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.276547 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3f8b396-45d6-403c-a699-9eb5d3d5c627-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-9ns75\" (UID: \"c3f8b396-45d6-403c-a699-9eb5d3d5c627\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9ns75" Jul 17 21:47:12.276769 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.276602 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-service-ca\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.276769 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.276645 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/632c3dc1-9978-4110-a20c-e32aa34a0e60-console-serving-cert\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.276769 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.276686 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmtd4\" (UniqueName: \"kubernetes.io/projected/c3f8b396-45d6-403c-a699-9eb5d3d5c627-kube-api-access-tmtd4\") pod \"monitoring-plugin-7b7b7bc8bf-9ns75\" (UID: \"c3f8b396-45d6-403c-a699-9eb5d3d5c627\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9ns75" Jul 17 21:47:12.276769 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.276709 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-console-config\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.276981 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.276785 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/632c3dc1-9978-4110-a20c-e32aa34a0e60-console-oauth-config\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.276981 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.276830 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n926s\" (UniqueName: \"kubernetes.io/projected/632c3dc1-9978-4110-a20c-e32aa34a0e60-kube-api-access-n926s\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.276981 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.276868 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-trusted-ca-bundle\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.276981 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.276894 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-oauth-serving-cert\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.277178 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:47:12.277046 2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 17 21:47:12.277178 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:47:12.277106 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3f8b396-45d6-403c-a699-9eb5d3d5c627-monitoring-plugin-cert podName:c3f8b396-45d6-403c-a699-9eb5d3d5c627 nodeName:}" failed. No retries permitted until 2026-07-17 21:47:12.777085778 +0000 UTC m=+69.637309272 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/c3f8b396-45d6-403c-a699-9eb5d3d5c627-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-9ns75" (UID: "c3f8b396-45d6-403c-a699-9eb5d3d5c627") : secret "monitoring-plugin-cert" not found Jul 17 21:47:12.288722 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.288674 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmtd4\" (UniqueName: \"kubernetes.io/projected/c3f8b396-45d6-403c-a699-9eb5d3d5c627-kube-api-access-tmtd4\") pod \"monitoring-plugin-7b7b7bc8bf-9ns75\" (UID: \"c3f8b396-45d6-403c-a699-9eb5d3d5c627\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9ns75" Jul 17 21:47:12.378635 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.378386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/632c3dc1-9978-4110-a20c-e32aa34a0e60-console-oauth-config\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.378635 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.378463 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n926s\" (UniqueName: \"kubernetes.io/projected/632c3dc1-9978-4110-a20c-e32aa34a0e60-kube-api-access-n926s\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.385796 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.378547 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-trusted-ca-bundle\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.385796 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.383636 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-oauth-serving-cert\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.385796 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.383734 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-service-ca\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.385796 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.383776 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/632c3dc1-9978-4110-a20c-e32aa34a0e60-console-serving-cert\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.385796 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.383824 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-console-config\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.385796 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.384634 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-console-config\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.385796 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.384960 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-service-ca\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.388852 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.388537 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-oauth-serving-cert\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.392326 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.392005 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-trusted-ca-bundle\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.397248 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.397175 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/632c3dc1-9978-4110-a20c-e32aa34a0e60-console-serving-cert\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.402235 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.402214 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/632c3dc1-9978-4110-a20c-e32aa34a0e60-console-oauth-config\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.403721 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.403675 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n926s\" (UniqueName: \"kubernetes.io/projected/632c3dc1-9978-4110-a20c-e32aa34a0e60-kube-api-access-n926s\") pod \"console-cd949f4fd-vsfjk\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.481291 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.481032 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:12.787970 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.787889 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3f8b396-45d6-403c-a699-9eb5d3d5c627-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-9ns75\" (UID: \"c3f8b396-45d6-403c-a699-9eb5d3d5c627\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9ns75" Jul 17 21:47:12.790991 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:12.790961 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c3f8b396-45d6-403c-a699-9eb5d3d5c627-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-9ns75\" (UID: \"c3f8b396-45d6-403c-a699-9eb5d3d5c627\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9ns75" Jul 17 21:47:13.071215 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:13.071133 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9ns75" Jul 17 21:47:16.430849 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:16.430789 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cd949f4fd-vsfjk"] Jul 17 21:47:16.444896 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:16.444870 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9ns75"] Jul 17 21:47:16.580305 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:16.580121 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-85f79f4d9-w98td"] Jul 17 21:47:16.585682 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:47:16.585632 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61876aa9_16d9_4016_b24c_f63fa89c32fb.slice/crio-9b9dfc81aa17d1b7611ec96e8e714e52ae3e6cfdaa3aa29c2f59bc415baf4a17 WatchSource:0}: Error finding container 9b9dfc81aa17d1b7611ec96e8e714e52ae3e6cfdaa3aa29c2f59bc415baf4a17: Status 404 returned error can't find the container with id 9b9dfc81aa17d1b7611ec96e8e714e52ae3e6cfdaa3aa29c2f59bc415baf4a17 Jul 17 21:47:17.075225 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:17.075181 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kp5bb" event={"ID":"ee877ecd-672d-4415-a69e-831da8e098fe","Type":"ContainerStarted","Data":"c9789b47a0471839eb358133465363b1c7a682febe08f1252cbfc1c2ace1a1d2"} Jul 17 21:47:17.076076 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:17.076050 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:47:17.080104 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:17.079219 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" event={"ID":"512670a9-3155-4a10-85c9-d4d4826d32e9","Type":"ContainerStarted","Data":"027288a32b1e05d0a682970e26ec27413ccb1accbd16a7855d83de543fde6588"} Jul 17 21:47:17.080104 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:17.079289 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" event={"ID":"512670a9-3155-4a10-85c9-d4d4826d32e9","Type":"ContainerStarted","Data":"a87aedc62f1052633e72ab5338ec881baa6eb0c4e6d7e340af694e7941ffa862"} Jul 17 21:47:17.080104 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:17.079331 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" event={"ID":"512670a9-3155-4a10-85c9-d4d4826d32e9","Type":"ContainerStarted","Data":"da3d67999c9f32ba9a9696a2299204348d461ada0d0a8a0d4f2ca3981d21ecfd"} Jul 17 21:47:17.082110 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:17.082085 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd949f4fd-vsfjk" event={"ID":"632c3dc1-9978-4110-a20c-e32aa34a0e60","Type":"ContainerStarted","Data":"f3e94d032c4ef949d2edfa8b369a9df1ba7cf22d8f26197688e95e0062ff058c"} Jul 17 21:47:17.082216 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:17.082132 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd949f4fd-vsfjk" event={"ID":"632c3dc1-9978-4110-a20c-e32aa34a0e60","Type":"ContainerStarted","Data":"e6dd7f837276115974bea5b006a0f3dd51978aa740b89c973974f90d5e24de00"} Jul 17 21:47:17.084223 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:17.084195 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" event={"ID":"61876aa9-16d9-4016-b24c-f63fa89c32fb","Type":"ContainerStarted","Data":"9b9dfc81aa17d1b7611ec96e8e714e52ae3e6cfdaa3aa29c2f59bc415baf4a17"} Jul 17 21:47:17.087633 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:17.087375 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"284136da-9486-4e1f-a654-ac4459450bbf","Type":"ContainerStarted","Data":"85eb80224e679cf8aa46cbf02a2410aee856df0b1b2114eef96d62e7954333ef"} Jul 17 21:47:17.087633 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:17.087442 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"284136da-9486-4e1f-a654-ac4459450bbf","Type":"ContainerStarted","Data":"697bf51e6925f9a64ca2b6ba884c46ec7297d312c6280bbf1d32e5cfafc5a36f"} Jul 17 21:47:17.087633 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:17.087480 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"284136da-9486-4e1f-a654-ac4459450bbf","Type":"ContainerStarted","Data":"3de4e2f0ba6c806b938c7ca4f6933d61c4634cea113bc82bcd48b66b80540bd8"} Jul 17 21:47:17.087633 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:17.087546 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"284136da-9486-4e1f-a654-ac4459450bbf","Type":"ContainerStarted","Data":"2131824fa2d8716bc6a1be422d288fb5bdcf454f6ebd2d0adc005fbea098e6af"} Jul 17 21:47:17.087633 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:17.087585 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"284136da-9486-4e1f-a654-ac4459450bbf","Type":"ContainerStarted","Data":"09f8a257b6523edfcf4c4011a48109c1e1017a25178abc18f2bf544a7f78c27c"} Jul 17 21:47:17.089440 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:17.089415 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9ns75" event={"ID":"c3f8b396-45d6-403c-a699-9eb5d3d5c627","Type":"ContainerStarted","Data":"cd60d47a6105e9e2cc78f639b849843a6ad55e9b980d9d3c597877a6b589d3ae"} Jul 17 21:47:17.093226 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:17.093174 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kp5bb" podStartSLOduration=67.092729659 podStartE2EDuration="1m13.093156552s" totalImagesPullingTime="6.000426893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:17.091580029 +0000 UTC m=+73.951803556" watchObservedRunningTime="2026-07-17 21:47:17.093156552 +0000 UTC m=+73.953380056" Jul 17 21:47:17.093340 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:17.093256 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-982gr" event={"ID":"e2430562-9210-4afd-9800-9e66f9548cb7","Type":"ContainerStarted","Data":"2f869b9f81b348fd545fda0cc9a1cc374f65d1ae7e811bdedfa8e35d0996b84f"} Jul 17 21:47:17.128321 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:17.128246 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cd949f4fd-vsfjk" podStartSLOduration=5.128225393 podStartE2EDuration="5.128225393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:17.111638999 +0000 UTC m=+73.971862504" watchObservedRunningTime="2026-07-17 21:47:17.128225393 +0000 UTC m=+73.988448890" Jul 17 21:47:17.128472 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:17.128440 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-982gr" podStartSLOduration=66.993348832 podStartE2EDuration="1m13.12842552s" totalImagesPullingTime="6.135076688s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:46:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:17.128126074 +0000 UTC m=+73.988349577" watchObservedRunningTime="2026-07-17 21:47:17.12842552 +0000 UTC m=+73.988649030" Jul 17 21:47:18.108302 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:18.108263 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"284136da-9486-4e1f-a654-ac4459450bbf","Type":"ContainerStarted","Data":"a57a3ea823e65663c793ab16cba93fb92431316834a9489db1e203d4baefa122"} Jul 17 21:47:18.111856 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:18.111825 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" event={"ID":"512670a9-3155-4a10-85c9-d4d4826d32e9","Type":"ContainerStarted","Data":"e45cdb7910eebad1c95b6cb29a2bdc3ff8fa8cd50c97284fca986647d458cfa1"} Jul 17 21:47:18.112017 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:18.111896 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" event={"ID":"512670a9-3155-4a10-85c9-d4d4826d32e9","Type":"ContainerStarted","Data":"9b75c8479a1af5aa56be0d3322f7b3c3ab5ffcee49fd3e8cbb5f77662b7f4f16"} Jul 17 21:47:18.112017 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:18.111931 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" event={"ID":"512670a9-3155-4a10-85c9-d4d4826d32e9","Type":"ContainerStarted","Data":"7b86d38106130576b0c827e03b30112fe02f6e336657cff2ed75a735523032c8"} Jul 17 21:47:18.136777 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:18.136704 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.272444858 podStartE2EDuration="10.136684244s" totalImagesPullingTime="6.843403372s" totalInitContainerRuntime="20.836014ms" isStatefulPod=false podCreationTimestamp="2026-07-17 21:47:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:18.133336946 +0000 UTC m=+74.993560449" watchObservedRunningTime="2026-07-17 21:47:18.136684244 +0000 UTC m=+74.996907758" Jul 17 21:47:18.163380 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:18.163285 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" podStartSLOduration=2.188037433 podStartE2EDuration="9.163260501s" totalImagesPullingTime="6.975223068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:47:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:18.160581154 +0000 UTC m=+75.020804678" watchObservedRunningTime="2026-07-17 21:47:18.163260501 +0000 UTC m=+75.023484006" Jul 17 21:47:19.116911 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:19.116873 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" event={"ID":"61876aa9-16d9-4016-b24c-f63fa89c32fb","Type":"ContainerStarted","Data":"3d37416970d48c7ccabd345c8334708d63cd412e2dcee8c39ad0b4d3db1a8d06"} Jul 17 21:47:19.118436 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:19.118408 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9ns75" event={"ID":"c3f8b396-45d6-403c-a699-9eb5d3d5c627","Type":"ContainerStarted","Data":"188f78c21363eb444d8bfd7b8a489de7d2027a24463c6dc3c9054964bbd2f6d4"} Jul 17 21:47:19.118562 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:19.118508 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:19.133773 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:19.133599 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" podStartSLOduration=6.461836525 podStartE2EDuration="8.133578751s" totalImagesPullingTime="1.671742226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:19.132662162 +0000 UTC m=+75.992885663" watchObservedRunningTime="2026-07-17 21:47:19.133578751 +0000 UTC m=+75.993802255" Jul 17 21:47:19.149704 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:19.149653 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9ns75" podStartSLOduration=5.340844068 podStartE2EDuration="7.149635672s" totalImagesPullingTime="1.808791604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:19.148062403 +0000 UTC m=+76.008285902" watchObservedRunningTime="2026-07-17 21:47:19.149635672 +0000 UTC m=+76.009859174" Jul 17 21:47:20.122692 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:20.122654 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9ns75" Jul 17 21:47:20.127532 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:20.127511 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-9ns75" Jul 17 21:47:22.481799 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:22.481761 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:22.482259 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:22.481814 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:22.486734 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:22.486714 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:23.137401 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:23.137369 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:47:23.187445 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:23.187418 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-676c555f45-smpsd"] Jul 17 21:47:25.130073 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:25.129949 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8666d66855-vl4vv" Jul 17 21:47:32.077112 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:32.077075 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:32.077595 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:32.077139 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:48.207733 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:48.207670 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-676c555f45-smpsd" podUID="9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728" containerName="console" containerID="cri-o://e1280e956af2c561d5fcefa0319ed1bca65c9cb10804eedc1c593c203ad0d2dc" gracePeriod=15 Jul 17 21:47:48.978604 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:48.978575 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:47:49.100893 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.100807 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-console-oauth-config\") pod \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " Jul 17 21:47:49.100893 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.100875 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-service-ca\") pod \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " Jul 17 21:47:49.101114 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.100902 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-console-serving-cert\") pod \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " Jul 17 21:47:49.101114 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.100992 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-trusted-ca-bundle\") pod \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " Jul 17 21:47:49.101114 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.101023 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-console-config\") pod \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " Jul 17 21:47:49.101114 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.101078 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhxmh\" (UniqueName: \"kubernetes.io/projected/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-kube-api-access-bhxmh\") pod \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " Jul 17 21:47:49.101114 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.101107 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-oauth-serving-cert\") pod \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\" (UID: \"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728\") " Jul 17 21:47:49.101380 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.101355 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728" (UID: "9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:47:49.101380 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.101369 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-service-ca" (OuterVolumeSpecName: "service-ca") pod "9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728" (UID: "9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:47:49.101601 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.101448 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-console-config" (OuterVolumeSpecName: "console-config") pod "9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728" (UID: "9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:47:49.101726 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.101621 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728" (UID: "9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:47:49.103391 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.103368 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728" (UID: "9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:47:49.103464 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.103382 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-kube-api-access-bhxmh" (OuterVolumeSpecName: "kube-api-access-bhxmh") pod "9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728" (UID: "9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728"). InnerVolumeSpecName "kube-api-access-bhxmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:47:49.103464 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.103432 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728" (UID: "9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:47:49.121478 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.121460 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kp5bb" Jul 17 21:47:49.202629 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.202591 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:47:49.202629 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.202621 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-console-serving-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:47:49.202629 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.202636 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-trusted-ca-bundle\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:47:49.202852 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.202648 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-console-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:47:49.202852 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.202661 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhxmh\" (UniqueName: \"kubernetes.io/projected/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-kube-api-access-bhxmh\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:47:49.202852 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.202672 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-oauth-serving-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:47:49.202852 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.202682 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728-console-oauth-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:47:49.245532 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.245478 2577 generic.go:356] "Generic (PLEG): container finished" podID="9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728" containerID="e1280e956af2c561d5fcefa0319ed1bca65c9cb10804eedc1c593c203ad0d2dc" exitCode=0 Jul 17 21:47:49.245969 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.245542 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-676c555f45-smpsd" Jul 17 21:47:49.245969 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.245587 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676c555f45-smpsd" event={"ID":"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728","Type":"ContainerDied","Data":"e1280e956af2c561d5fcefa0319ed1bca65c9cb10804eedc1c593c203ad0d2dc"} Jul 17 21:47:49.245969 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.245649 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-676c555f45-smpsd" event={"ID":"9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728","Type":"ContainerDied","Data":"ec0f50e7d9e02942f05c3aaf168360764163cd38863974d2acb7d5731afc88a8"} Jul 17 21:47:49.245969 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.245681 2577 scope.go:117] "RemoveContainer" containerID="e1280e956af2c561d5fcefa0319ed1bca65c9cb10804eedc1c593c203ad0d2dc" Jul 17 21:47:49.256593 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.256573 2577 scope.go:117] "RemoveContainer" containerID="e1280e956af2c561d5fcefa0319ed1bca65c9cb10804eedc1c593c203ad0d2dc" Jul 17 21:47:49.256847 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:47:49.256826 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1280e956af2c561d5fcefa0319ed1bca65c9cb10804eedc1c593c203ad0d2dc\": container with ID starting with e1280e956af2c561d5fcefa0319ed1bca65c9cb10804eedc1c593c203ad0d2dc not found: ID does not exist" containerID="e1280e956af2c561d5fcefa0319ed1bca65c9cb10804eedc1c593c203ad0d2dc" Jul 17 21:47:49.256901 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.256857 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1280e956af2c561d5fcefa0319ed1bca65c9cb10804eedc1c593c203ad0d2dc"} err="failed to get container status \"e1280e956af2c561d5fcefa0319ed1bca65c9cb10804eedc1c593c203ad0d2dc\": rpc error: code = NotFound desc = could not find container \"e1280e956af2c561d5fcefa0319ed1bca65c9cb10804eedc1c593c203ad0d2dc\": container with ID starting with e1280e956af2c561d5fcefa0319ed1bca65c9cb10804eedc1c593c203ad0d2dc not found: ID does not exist" Jul 17 21:47:49.269601 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.269045 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-676c555f45-smpsd"] Jul 17 21:47:49.273122 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.273088 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-676c555f45-smpsd"] Jul 17 21:47:49.699366 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:49.699331 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728" path="/var/lib/kubelet/pods/9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728/volumes" Jul 17 21:47:52.082295 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:52.082268 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:47:52.085983 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:47:52.085961 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-85f79f4d9-w98td" Jul 17 21:48:31.684270 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.684236 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g"] Jul 17 21:48:31.684907 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.684885 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728" containerName="console" Jul 17 21:48:31.684993 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.684911 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728" containerName="console" Jul 17 21:48:31.684993 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.684988 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b5eeaf8-bfdc-4e12-98ab-7e0ab8fcf728" containerName="console" Jul 17 21:48:31.687531 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.687509 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.690801 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.690774 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 17 21:48:31.691329 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.691303 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 17 21:48:31.691431 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.691351 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 17 21:48:31.691605 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.691447 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-ptjxg\"" Jul 17 21:48:31.691747 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.691658 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 17 21:48:31.691817 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.691766 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 17 21:48:31.696228 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.696207 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 17 21:48:31.700978 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.700957 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g"] Jul 17 21:48:31.848282 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.848247 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.848437 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.848304 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-secret-telemeter-client\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.848510 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.848459 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.848572 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.848553 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mbrq\" (UniqueName: \"kubernetes.io/projected/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-kube-api-access-7mbrq\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.848612 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.848598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-metrics-client-ca\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.848673 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.848657 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-federate-client-tls\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.848758 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.848736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-telemeter-client-tls\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.848831 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.848816 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-serving-certs-ca-bundle\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.950004 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.949978 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.950146 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.950015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-secret-telemeter-client\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.950146 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.950041 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.950146 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.950075 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mbrq\" (UniqueName: \"kubernetes.io/projected/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-kube-api-access-7mbrq\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.950146 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.950095 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-metrics-client-ca\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.950146 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.950115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-federate-client-tls\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.950375 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.950151 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-telemeter-client-tls\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.950375 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.950191 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-serving-certs-ca-bundle\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.951087 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.951017 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.951234 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.951136 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-serving-certs-ca-bundle\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.951234 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.951201 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-metrics-client-ca\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.952711 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.952683 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.952875 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.952857 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-telemeter-client-tls\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.953052 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.953034 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-secret-telemeter-client\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.953239 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.953217 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-federate-client-tls\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.957379 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.957357 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mbrq\" (UniqueName: \"kubernetes.io/projected/0d19c18f-a526-4fb4-9538-59cd7aa25aa7-kube-api-access-7mbrq\") pod \"telemeter-client-6c54c9c77f-ckk9g\" (UID: \"0d19c18f-a526-4fb4-9538-59cd7aa25aa7\") " pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:31.998524 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:31.998502 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" Jul 17 21:48:32.132980 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:32.132783 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g"] Jul 17 21:48:32.135830 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:48:32.135806 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d19c18f_a526_4fb4_9538_59cd7aa25aa7.slice/crio-54f0231c2b2dd88eba89e36d3b19b2eb0886abb1bb377adc6ab7c83454c42d6a WatchSource:0}: Error finding container 54f0231c2b2dd88eba89e36d3b19b2eb0886abb1bb377adc6ab7c83454c42d6a: Status 404 returned error can't find the container with id 54f0231c2b2dd88eba89e36d3b19b2eb0886abb1bb377adc6ab7c83454c42d6a Jul 17 21:48:32.412670 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:32.412572 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" event={"ID":"0d19c18f-a526-4fb4-9538-59cd7aa25aa7","Type":"ContainerStarted","Data":"54f0231c2b2dd88eba89e36d3b19b2eb0886abb1bb377adc6ab7c83454c42d6a"} Jul 17 21:48:34.423535 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:34.423427 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" event={"ID":"0d19c18f-a526-4fb4-9538-59cd7aa25aa7","Type":"ContainerStarted","Data":"e57f66074dc5a41647290839a7fb418083b0e8982855180c1e1fde8667cf6691"} Jul 17 21:48:34.424078 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:34.423546 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" event={"ID":"0d19c18f-a526-4fb4-9538-59cd7aa25aa7","Type":"ContainerStarted","Data":"e686fb2aa7ac89e0eed63cf2c86120fc9d1600fcd4aa8581ca14f63df4361d75"} Jul 17 21:48:34.424078 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:34.423574 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" event={"ID":"0d19c18f-a526-4fb4-9538-59cd7aa25aa7","Type":"ContainerStarted","Data":"328874a55aae6b60a38d774643e97747a7599d23eb073ad8c823c7f1c98b4386"} Jul 17 21:48:34.444735 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:34.444686 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6c54c9c77f-ckk9g" podStartSLOduration=1.4966078729999999 podStartE2EDuration="3.444666997s" totalImagesPullingTime="1.948059124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:48:34.443472966 +0000 UTC m=+151.303696516" watchObservedRunningTime="2026-07-17 21:48:34.444666997 +0000 UTC m=+151.304890499" Jul 17 21:48:35.078807 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.078770 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6f5647f5d9-rwpng"] Jul 17 21:48:35.081644 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.081624 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.098797 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.098767 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f5647f5d9-rwpng"] Jul 17 21:48:35.180377 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.180339 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f561691f-6198-47e1-a226-342bae496231-console-serving-cert\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.180377 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.180380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-trusted-ca-bundle\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.180665 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.180405 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-console-config\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.180665 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.180511 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f561691f-6198-47e1-a226-342bae496231-console-oauth-config\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.180665 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.180547 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-service-ca\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.180665 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.180570 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76fxq\" (UniqueName: \"kubernetes.io/projected/f561691f-6198-47e1-a226-342bae496231-kube-api-access-76fxq\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.180665 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.180624 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-oauth-serving-cert\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.281430 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.281397 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f561691f-6198-47e1-a226-342bae496231-console-serving-cert\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.281430 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.281430 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-trusted-ca-bundle\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.281702 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.281450 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-console-config\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.281702 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.281475 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f561691f-6198-47e1-a226-342bae496231-console-oauth-config\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.281702 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.281517 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-service-ca\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.281702 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.281535 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76fxq\" (UniqueName: \"kubernetes.io/projected/f561691f-6198-47e1-a226-342bae496231-kube-api-access-76fxq\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.281702 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.281574 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-oauth-serving-cert\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.282318 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.282288 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-service-ca\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.282437 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.282364 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-trusted-ca-bundle\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.282437 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.282386 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-console-config\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.282589 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.282568 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-oauth-serving-cert\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.284038 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.284012 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f561691f-6198-47e1-a226-342bae496231-console-serving-cert\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.284135 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.284062 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f561691f-6198-47e1-a226-342bae496231-console-oauth-config\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.289639 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.289618 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76fxq\" (UniqueName: \"kubernetes.io/projected/f561691f-6198-47e1-a226-342bae496231-kube-api-access-76fxq\") pod \"console-6f5647f5d9-rwpng\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.392263 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.392188 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:35.520931 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:35.520902 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6f5647f5d9-rwpng"] Jul 17 21:48:35.523669 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:48:35.523636 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf561691f_6198_47e1_a226_342bae496231.slice/crio-3497edb893ca6869607685e8e3976f7d345237bbdf3edd549281829ec75c10a7 WatchSource:0}: Error finding container 3497edb893ca6869607685e8e3976f7d345237bbdf3edd549281829ec75c10a7: Status 404 returned error can't find the container with id 3497edb893ca6869607685e8e3976f7d345237bbdf3edd549281829ec75c10a7 Jul 17 21:48:36.432969 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:36.432932 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5647f5d9-rwpng" event={"ID":"f561691f-6198-47e1-a226-342bae496231","Type":"ContainerStarted","Data":"8c7da57aff28d9e5ce32dc657e00a6090e73947cdfaf1520f1fc8276079da40b"} Jul 17 21:48:36.433166 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:36.432980 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5647f5d9-rwpng" event={"ID":"f561691f-6198-47e1-a226-342bae496231","Type":"ContainerStarted","Data":"3497edb893ca6869607685e8e3976f7d345237bbdf3edd549281829ec75c10a7"} Jul 17 21:48:36.451621 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:36.451574 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6f5647f5d9-rwpng" podStartSLOduration=1.4515595079999999 podStartE2EDuration="1.451559508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:48:36.450070066 +0000 UTC m=+153.310293627" watchObservedRunningTime="2026-07-17 21:48:36.451559508 +0000 UTC m=+153.311783010" Jul 17 21:48:45.392613 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.392577 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:45.392981 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.392656 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:45.398192 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.398170 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:45.470225 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.470198 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:48:45.521666 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.521634 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cd949f4fd-vsfjk"] Jul 17 21:48:45.607512 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.607184 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65fffd6797-j4mqh"] Jul 17 21:48:45.614852 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.614820 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.618104 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.618072 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65fffd6797-j4mqh"] Jul 17 21:48:45.674962 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.674891 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/930695d4-4edb-4c35-bdeb-6b90356ba153-console-serving-cert\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.674962 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.674922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-console-config\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.675126 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.674967 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbg2b\" (UniqueName: \"kubernetes.io/projected/930695d4-4edb-4c35-bdeb-6b90356ba153-kube-api-access-wbg2b\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.675126 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.674990 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-oauth-serving-cert\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.675126 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.675007 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-trusted-ca-bundle\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.675126 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.675025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/930695d4-4edb-4c35-bdeb-6b90356ba153-console-oauth-config\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.675126 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.675102 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-service-ca\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.775996 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.775960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbg2b\" (UniqueName: \"kubernetes.io/projected/930695d4-4edb-4c35-bdeb-6b90356ba153-kube-api-access-wbg2b\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.775996 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.775999 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-oauth-serving-cert\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.776234 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.776027 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-trusted-ca-bundle\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.776234 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.776053 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/930695d4-4edb-4c35-bdeb-6b90356ba153-console-oauth-config\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.776234 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.776081 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-service-ca\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.776234 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.776139 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/930695d4-4edb-4c35-bdeb-6b90356ba153-console-serving-cert\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.776447 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.776279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-console-config\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.776861 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.776832 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-oauth-serving-cert\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.776990 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.776872 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-service-ca\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.776990 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.776899 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-trusted-ca-bundle\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.777172 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.777150 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-console-config\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.778634 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.778610 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/930695d4-4edb-4c35-bdeb-6b90356ba153-console-serving-cert\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.778726 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.778659 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/930695d4-4edb-4c35-bdeb-6b90356ba153-console-oauth-config\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.784886 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.784868 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbg2b\" (UniqueName: \"kubernetes.io/projected/930695d4-4edb-4c35-bdeb-6b90356ba153-kube-api-access-wbg2b\") pod \"console-65fffd6797-j4mqh\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:45.926121 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:45.926033 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:46.066885 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:46.066855 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65fffd6797-j4mqh"] Jul 17 21:48:46.070921 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:48:46.070898 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930695d4_4edb_4c35_bdeb_6b90356ba153.slice/crio-4ae8da4cf33d36d606b3e4ab28633249a7fa906bc686668087ac47ffafad4afc WatchSource:0}: Error finding container 4ae8da4cf33d36d606b3e4ab28633249a7fa906bc686668087ac47ffafad4afc: Status 404 returned error can't find the container with id 4ae8da4cf33d36d606b3e4ab28633249a7fa906bc686668087ac47ffafad4afc Jul 17 21:48:46.471945 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:46.471907 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fffd6797-j4mqh" event={"ID":"930695d4-4edb-4c35-bdeb-6b90356ba153","Type":"ContainerStarted","Data":"f330d92bc530f1f23e7218044cdf7fc6be81329fe8b9f4c74ef5829a0881ae5d"} Jul 17 21:48:46.472345 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:46.471961 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fffd6797-j4mqh" event={"ID":"930695d4-4edb-4c35-bdeb-6b90356ba153","Type":"ContainerStarted","Data":"4ae8da4cf33d36d606b3e4ab28633249a7fa906bc686668087ac47ffafad4afc"} Jul 17 21:48:55.926606 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:55.926523 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:55.926606 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:55.926578 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:55.931301 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:55.931278 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:55.949997 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:55.949946 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65fffd6797-j4mqh" podStartSLOduration=10.949922072 podStartE2EDuration="10.949922072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:48:46.492946778 +0000 UTC m=+163.353170279" watchObservedRunningTime="2026-07-17 21:48:55.949922072 +0000 UTC m=+172.810145575" Jul 17 21:48:56.519756 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:56.519726 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:48:56.565870 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:48:56.565842 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f5647f5d9-rwpng"] Jul 17 21:49:10.542744 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:10.542691 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cd949f4fd-vsfjk" podUID="632c3dc1-9978-4110-a20c-e32aa34a0e60" containerName="console" containerID="cri-o://f3e94d032c4ef949d2edfa8b369a9df1ba7cf22d8f26197688e95e0062ff058c" gracePeriod=15 Jul 17 21:49:10.788670 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:10.788644 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:49:10.906199 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:10.906121 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n926s\" (UniqueName: \"kubernetes.io/projected/632c3dc1-9978-4110-a20c-e32aa34a0e60-kube-api-access-n926s\") pod \"632c3dc1-9978-4110-a20c-e32aa34a0e60\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " Jul 17 21:49:10.906199 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:10.906182 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-service-ca\") pod \"632c3dc1-9978-4110-a20c-e32aa34a0e60\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " Jul 17 21:49:10.906415 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:10.906205 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/632c3dc1-9978-4110-a20c-e32aa34a0e60-console-oauth-config\") pod \"632c3dc1-9978-4110-a20c-e32aa34a0e60\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " Jul 17 21:49:10.906415 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:10.906245 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-oauth-serving-cert\") pod \"632c3dc1-9978-4110-a20c-e32aa34a0e60\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " Jul 17 21:49:10.906415 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:10.906272 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-trusted-ca-bundle\") pod \"632c3dc1-9978-4110-a20c-e32aa34a0e60\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " Jul 17 21:49:10.906415 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:10.906305 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/632c3dc1-9978-4110-a20c-e32aa34a0e60-console-serving-cert\") pod \"632c3dc1-9978-4110-a20c-e32aa34a0e60\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " Jul 17 21:49:10.906415 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:10.906328 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-console-config\") pod \"632c3dc1-9978-4110-a20c-e32aa34a0e60\" (UID: \"632c3dc1-9978-4110-a20c-e32aa34a0e60\") " Jul 17 21:49:10.906764 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:10.906738 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "632c3dc1-9978-4110-a20c-e32aa34a0e60" (UID: "632c3dc1-9978-4110-a20c-e32aa34a0e60"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:49:10.906836 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:10.906735 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-service-ca" (OuterVolumeSpecName: "service-ca") pod "632c3dc1-9978-4110-a20c-e32aa34a0e60" (UID: "632c3dc1-9978-4110-a20c-e32aa34a0e60"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:49:10.906836 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:10.906821 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "632c3dc1-9978-4110-a20c-e32aa34a0e60" (UID: "632c3dc1-9978-4110-a20c-e32aa34a0e60"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:49:10.907101 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:10.906975 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-console-config" (OuterVolumeSpecName: "console-config") pod "632c3dc1-9978-4110-a20c-e32aa34a0e60" (UID: "632c3dc1-9978-4110-a20c-e32aa34a0e60"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:49:10.908603 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:10.908583 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/632c3dc1-9978-4110-a20c-e32aa34a0e60-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "632c3dc1-9978-4110-a20c-e32aa34a0e60" (UID: "632c3dc1-9978-4110-a20c-e32aa34a0e60"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:49:10.908685 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:10.908635 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/632c3dc1-9978-4110-a20c-e32aa34a0e60-kube-api-access-n926s" (OuterVolumeSpecName: "kube-api-access-n926s") pod "632c3dc1-9978-4110-a20c-e32aa34a0e60" (UID: "632c3dc1-9978-4110-a20c-e32aa34a0e60"). InnerVolumeSpecName "kube-api-access-n926s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:49:10.908824 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:10.908801 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/632c3dc1-9978-4110-a20c-e32aa34a0e60-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "632c3dc1-9978-4110-a20c-e32aa34a0e60" (UID: "632c3dc1-9978-4110-a20c-e32aa34a0e60"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:49:11.007759 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:11.007728 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:49:11.007759 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:11.007754 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/632c3dc1-9978-4110-a20c-e32aa34a0e60-console-oauth-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:49:11.007759 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:11.007764 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-oauth-serving-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:49:11.007959 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:11.007773 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-trusted-ca-bundle\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:49:11.007959 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:11.007781 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/632c3dc1-9978-4110-a20c-e32aa34a0e60-console-serving-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:49:11.007959 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:11.007789 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/632c3dc1-9978-4110-a20c-e32aa34a0e60-console-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:49:11.007959 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:11.007798 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n926s\" (UniqueName: \"kubernetes.io/projected/632c3dc1-9978-4110-a20c-e32aa34a0e60-kube-api-access-n926s\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:49:11.577635 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:11.577596 2577 generic.go:356] "Generic (PLEG): container finished" podID="632c3dc1-9978-4110-a20c-e32aa34a0e60" containerID="f3e94d032c4ef949d2edfa8b369a9df1ba7cf22d8f26197688e95e0062ff058c" exitCode=0 Jul 17 21:49:11.578022 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:11.577643 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cd949f4fd-vsfjk" Jul 17 21:49:11.578022 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:11.577668 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd949f4fd-vsfjk" event={"ID":"632c3dc1-9978-4110-a20c-e32aa34a0e60","Type":"ContainerDied","Data":"f3e94d032c4ef949d2edfa8b369a9df1ba7cf22d8f26197688e95e0062ff058c"} Jul 17 21:49:11.578022 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:11.577707 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cd949f4fd-vsfjk" event={"ID":"632c3dc1-9978-4110-a20c-e32aa34a0e60","Type":"ContainerDied","Data":"e6dd7f837276115974bea5b006a0f3dd51978aa740b89c973974f90d5e24de00"} Jul 17 21:49:11.578022 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:11.577729 2577 scope.go:117] "RemoveContainer" containerID="f3e94d032c4ef949d2edfa8b369a9df1ba7cf22d8f26197688e95e0062ff058c" Jul 17 21:49:11.586335 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:11.586319 2577 scope.go:117] "RemoveContainer" containerID="f3e94d032c4ef949d2edfa8b369a9df1ba7cf22d8f26197688e95e0062ff058c" Jul 17 21:49:11.586602 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:49:11.586585 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3e94d032c4ef949d2edfa8b369a9df1ba7cf22d8f26197688e95e0062ff058c\": container with ID starting with f3e94d032c4ef949d2edfa8b369a9df1ba7cf22d8f26197688e95e0062ff058c not found: ID does not exist" containerID="f3e94d032c4ef949d2edfa8b369a9df1ba7cf22d8f26197688e95e0062ff058c" Jul 17 21:49:11.586658 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:11.586610 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3e94d032c4ef949d2edfa8b369a9df1ba7cf22d8f26197688e95e0062ff058c"} err="failed to get container status \"f3e94d032c4ef949d2edfa8b369a9df1ba7cf22d8f26197688e95e0062ff058c\": rpc error: code = NotFound desc = could not find container \"f3e94d032c4ef949d2edfa8b369a9df1ba7cf22d8f26197688e95e0062ff058c\": container with ID starting with f3e94d032c4ef949d2edfa8b369a9df1ba7cf22d8f26197688e95e0062ff058c not found: ID does not exist" Jul 17 21:49:11.598749 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:11.598720 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cd949f4fd-vsfjk"] Jul 17 21:49:11.603914 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:11.603641 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cd949f4fd-vsfjk"] Jul 17 21:49:11.698906 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:11.698875 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="632c3dc1-9978-4110-a20c-e32aa34a0e60" path="/var/lib/kubelet/pods/632c3dc1-9978-4110-a20c-e32aa34a0e60/volumes" Jul 17 21:49:21.591393 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:21.591350 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6f5647f5d9-rwpng" podUID="f561691f-6198-47e1-a226-342bae496231" containerName="console" containerID="cri-o://8c7da57aff28d9e5ce32dc657e00a6090e73947cdfaf1520f1fc8276079da40b" gracePeriod=15 Jul 17 21:49:21.832451 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:21.832429 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:49:21.904952 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:21.904864 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f561691f-6198-47e1-a226-342bae496231-console-serving-cert\") pod \"f561691f-6198-47e1-a226-342bae496231\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " Jul 17 21:49:21.904952 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:21.904910 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-console-config\") pod \"f561691f-6198-47e1-a226-342bae496231\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " Jul 17 21:49:21.904952 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:21.904929 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-oauth-serving-cert\") pod \"f561691f-6198-47e1-a226-342bae496231\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " Jul 17 21:49:21.905228 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:21.904961 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-76fxq\" (UniqueName: \"kubernetes.io/projected/f561691f-6198-47e1-a226-342bae496231-kube-api-access-76fxq\") pod \"f561691f-6198-47e1-a226-342bae496231\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " Jul 17 21:49:21.905228 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:21.904998 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-service-ca\") pod \"f561691f-6198-47e1-a226-342bae496231\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " Jul 17 21:49:21.905228 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:21.905018 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f561691f-6198-47e1-a226-342bae496231-console-oauth-config\") pod \"f561691f-6198-47e1-a226-342bae496231\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " Jul 17 21:49:21.905228 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:21.905060 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-trusted-ca-bundle\") pod \"f561691f-6198-47e1-a226-342bae496231\" (UID: \"f561691f-6198-47e1-a226-342bae496231\") " Jul 17 21:49:21.905562 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:21.905522 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-console-config" (OuterVolumeSpecName: "console-config") pod "f561691f-6198-47e1-a226-342bae496231" (UID: "f561691f-6198-47e1-a226-342bae496231"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:49:21.905562 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:21.905552 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f561691f-6198-47e1-a226-342bae496231" (UID: "f561691f-6198-47e1-a226-342bae496231"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:49:21.905757 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:21.905636 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-service-ca" (OuterVolumeSpecName: "service-ca") pod "f561691f-6198-47e1-a226-342bae496231" (UID: "f561691f-6198-47e1-a226-342bae496231"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:49:21.905757 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:21.905687 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f561691f-6198-47e1-a226-342bae496231" (UID: "f561691f-6198-47e1-a226-342bae496231"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:49:21.907237 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:21.907215 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f561691f-6198-47e1-a226-342bae496231-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f561691f-6198-47e1-a226-342bae496231" (UID: "f561691f-6198-47e1-a226-342bae496231"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:49:21.907733 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:21.907711 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f561691f-6198-47e1-a226-342bae496231-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f561691f-6198-47e1-a226-342bae496231" (UID: "f561691f-6198-47e1-a226-342bae496231"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:49:21.907825 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:21.907804 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f561691f-6198-47e1-a226-342bae496231-kube-api-access-76fxq" (OuterVolumeSpecName: "kube-api-access-76fxq") pod "f561691f-6198-47e1-a226-342bae496231" (UID: "f561691f-6198-47e1-a226-342bae496231"). InnerVolumeSpecName "kube-api-access-76fxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:49:22.006439 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:22.006411 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:49:22.006439 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:22.006433 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f561691f-6198-47e1-a226-342bae496231-console-oauth-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:49:22.006439 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:22.006442 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-trusted-ca-bundle\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:49:22.006653 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:22.006450 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f561691f-6198-47e1-a226-342bae496231-console-serving-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:49:22.006653 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:22.006458 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-console-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:49:22.006653 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:22.006467 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f561691f-6198-47e1-a226-342bae496231-oauth-serving-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:49:22.006653 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:22.006475 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76fxq\" (UniqueName: \"kubernetes.io/projected/f561691f-6198-47e1-a226-342bae496231-kube-api-access-76fxq\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:49:22.622259 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:22.622219 2577 generic.go:356] "Generic (PLEG): container finished" podID="f561691f-6198-47e1-a226-342bae496231" containerID="8c7da57aff28d9e5ce32dc657e00a6090e73947cdfaf1520f1fc8276079da40b" exitCode=0 Jul 17 21:49:22.622741 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:22.622268 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6f5647f5d9-rwpng" Jul 17 21:49:22.622741 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:22.622293 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5647f5d9-rwpng" event={"ID":"f561691f-6198-47e1-a226-342bae496231","Type":"ContainerDied","Data":"8c7da57aff28d9e5ce32dc657e00a6090e73947cdfaf1520f1fc8276079da40b"} Jul 17 21:49:22.622741 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:22.622351 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6f5647f5d9-rwpng" event={"ID":"f561691f-6198-47e1-a226-342bae496231","Type":"ContainerDied","Data":"3497edb893ca6869607685e8e3976f7d345237bbdf3edd549281829ec75c10a7"} Jul 17 21:49:22.622741 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:22.622379 2577 scope.go:117] "RemoveContainer" containerID="8c7da57aff28d9e5ce32dc657e00a6090e73947cdfaf1520f1fc8276079da40b" Jul 17 21:49:22.635422 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:22.635405 2577 scope.go:117] "RemoveContainer" containerID="8c7da57aff28d9e5ce32dc657e00a6090e73947cdfaf1520f1fc8276079da40b" Jul 17 21:49:22.635717 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:49:22.635693 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c7da57aff28d9e5ce32dc657e00a6090e73947cdfaf1520f1fc8276079da40b\": container with ID starting with 8c7da57aff28d9e5ce32dc657e00a6090e73947cdfaf1520f1fc8276079da40b not found: ID does not exist" containerID="8c7da57aff28d9e5ce32dc657e00a6090e73947cdfaf1520f1fc8276079da40b" Jul 17 21:49:22.635802 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:22.635729 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7da57aff28d9e5ce32dc657e00a6090e73947cdfaf1520f1fc8276079da40b"} err="failed to get container status \"8c7da57aff28d9e5ce32dc657e00a6090e73947cdfaf1520f1fc8276079da40b\": rpc error: code = NotFound desc = could not find container \"8c7da57aff28d9e5ce32dc657e00a6090e73947cdfaf1520f1fc8276079da40b\": container with ID starting with 8c7da57aff28d9e5ce32dc657e00a6090e73947cdfaf1520f1fc8276079da40b not found: ID does not exist" Jul 17 21:49:22.646049 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:22.645469 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6f5647f5d9-rwpng"] Jul 17 21:49:22.647386 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:22.647367 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6f5647f5d9-rwpng"] Jul 17 21:49:23.698761 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:23.698732 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f561691f-6198-47e1-a226-342bae496231" path="/var/lib/kubelet/pods/f561691f-6198-47e1-a226-342bae496231/volumes" Jul 17 21:49:29.765252 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.765210 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j"] Jul 17 21:49:29.765738 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.765692 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f561691f-6198-47e1-a226-342bae496231" containerName="console" Jul 17 21:49:29.765738 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.765710 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f561691f-6198-47e1-a226-342bae496231" containerName="console" Jul 17 21:49:29.765738 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.765730 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="632c3dc1-9978-4110-a20c-e32aa34a0e60" containerName="console" Jul 17 21:49:29.765895 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.765741 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="632c3dc1-9978-4110-a20c-e32aa34a0e60" containerName="console" Jul 17 21:49:29.765895 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.765820 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f561691f-6198-47e1-a226-342bae496231" containerName="console" Jul 17 21:49:29.765895 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.765836 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="632c3dc1-9978-4110-a20c-e32aa34a0e60" containerName="console" Jul 17 21:49:29.770068 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.770048 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j" Jul 17 21:49:29.772400 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.772376 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 17 21:49:29.772513 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.772376 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 17 21:49:29.773553 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.773533 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 17 21:49:29.773553 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.773551 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-c9gbm\"" Jul 17 21:49:29.773728 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.773586 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 17 21:49:29.778189 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.778167 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j"] Jul 17 21:49:29.847073 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.847051 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b44654799-s8xjj"] Jul 17 21:49:29.850154 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.850140 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b44654799-s8xjj" Jul 17 21:49:29.852679 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.852661 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 17 21:49:29.858730 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.858709 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b44654799-s8xjj"] Jul 17 21:49:29.870856 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.870836 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f66983bc-fe53-4298-82af-f3a16cb256b3-tmp\") pod \"klusterlet-addon-workmgr-6b44654799-s8xjj\" (UID: \"f66983bc-fe53-4298-82af-f3a16cb256b3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b44654799-s8xjj" Jul 17 21:49:29.870954 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.870863 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f66983bc-fe53-4298-82af-f3a16cb256b3-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b44654799-s8xjj\" (UID: \"f66983bc-fe53-4298-82af-f3a16cb256b3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b44654799-s8xjj" Jul 17 21:49:29.870954 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.870924 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1dbe15a0-0d75-4f13-98c7-5ca2510aad39-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j\" (UID: \"1dbe15a0-0d75-4f13-98c7-5ca2510aad39\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j" Jul 17 21:49:29.870954 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.870943 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9n2k\" (UniqueName: \"kubernetes.io/projected/1dbe15a0-0d75-4f13-98c7-5ca2510aad39-kube-api-access-c9n2k\") pod \"managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j\" (UID: \"1dbe15a0-0d75-4f13-98c7-5ca2510aad39\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j" Jul 17 21:49:29.871095 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.870966 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ns6h\" (UniqueName: \"kubernetes.io/projected/f66983bc-fe53-4298-82af-f3a16cb256b3-kube-api-access-2ns6h\") pod \"klusterlet-addon-workmgr-6b44654799-s8xjj\" (UID: \"f66983bc-fe53-4298-82af-f3a16cb256b3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b44654799-s8xjj" Jul 17 21:49:29.971552 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.971519 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1dbe15a0-0d75-4f13-98c7-5ca2510aad39-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j\" (UID: \"1dbe15a0-0d75-4f13-98c7-5ca2510aad39\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j" Jul 17 21:49:29.971710 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.971558 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9n2k\" (UniqueName: \"kubernetes.io/projected/1dbe15a0-0d75-4f13-98c7-5ca2510aad39-kube-api-access-c9n2k\") pod \"managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j\" (UID: \"1dbe15a0-0d75-4f13-98c7-5ca2510aad39\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j" Jul 17 21:49:29.971710 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.971590 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ns6h\" (UniqueName: \"kubernetes.io/projected/f66983bc-fe53-4298-82af-f3a16cb256b3-kube-api-access-2ns6h\") pod \"klusterlet-addon-workmgr-6b44654799-s8xjj\" (UID: \"f66983bc-fe53-4298-82af-f3a16cb256b3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b44654799-s8xjj" Jul 17 21:49:29.971710 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.971656 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f66983bc-fe53-4298-82af-f3a16cb256b3-tmp\") pod \"klusterlet-addon-workmgr-6b44654799-s8xjj\" (UID: \"f66983bc-fe53-4298-82af-f3a16cb256b3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b44654799-s8xjj" Jul 17 21:49:29.971710 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.971678 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f66983bc-fe53-4298-82af-f3a16cb256b3-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b44654799-s8xjj\" (UID: \"f66983bc-fe53-4298-82af-f3a16cb256b3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b44654799-s8xjj" Jul 17 21:49:29.972014 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.971994 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f66983bc-fe53-4298-82af-f3a16cb256b3-tmp\") pod \"klusterlet-addon-workmgr-6b44654799-s8xjj\" (UID: \"f66983bc-fe53-4298-82af-f3a16cb256b3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b44654799-s8xjj" Jul 17 21:49:29.974141 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.974113 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1dbe15a0-0d75-4f13-98c7-5ca2510aad39-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j\" (UID: \"1dbe15a0-0d75-4f13-98c7-5ca2510aad39\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j" Jul 17 21:49:29.974291 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.974270 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f66983bc-fe53-4298-82af-f3a16cb256b3-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b44654799-s8xjj\" (UID: \"f66983bc-fe53-4298-82af-f3a16cb256b3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b44654799-s8xjj" Jul 17 21:49:29.980582 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.980558 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ns6h\" (UniqueName: \"kubernetes.io/projected/f66983bc-fe53-4298-82af-f3a16cb256b3-kube-api-access-2ns6h\") pod \"klusterlet-addon-workmgr-6b44654799-s8xjj\" (UID: \"f66983bc-fe53-4298-82af-f3a16cb256b3\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b44654799-s8xjj" Jul 17 21:49:29.980687 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:29.980657 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9n2k\" (UniqueName: \"kubernetes.io/projected/1dbe15a0-0d75-4f13-98c7-5ca2510aad39-kube-api-access-c9n2k\") pod \"managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j\" (UID: \"1dbe15a0-0d75-4f13-98c7-5ca2510aad39\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j" Jul 17 21:49:30.093515 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:30.093420 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j" Jul 17 21:49:30.160323 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:30.160298 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b44654799-s8xjj" Jul 17 21:49:30.229929 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:30.229901 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j"] Jul 17 21:49:30.232583 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:49:30.232546 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dbe15a0_0d75_4f13_98c7_5ca2510aad39.slice/crio-89a62119473196463cd9214a0eb22815336e4d939ff2c791c33f4ce0b92f78c5 WatchSource:0}: Error finding container 89a62119473196463cd9214a0eb22815336e4d939ff2c791c33f4ce0b92f78c5: Status 404 returned error can't find the container with id 89a62119473196463cd9214a0eb22815336e4d939ff2c791c33f4ce0b92f78c5 Jul 17 21:49:30.290721 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:30.290693 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b44654799-s8xjj"] Jul 17 21:49:30.293412 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:49:30.293389 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf66983bc_fe53_4298_82af_f3a16cb256b3.slice/crio-e53952d01b4942121859cb53fdfdcc23afad8ca55df1a6768a2ed49e7f244654 WatchSource:0}: Error finding container e53952d01b4942121859cb53fdfdcc23afad8ca55df1a6768a2ed49e7f244654: Status 404 returned error can't find the container with id e53952d01b4942121859cb53fdfdcc23afad8ca55df1a6768a2ed49e7f244654 Jul 17 21:49:30.656308 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:30.656275 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b44654799-s8xjj" event={"ID":"f66983bc-fe53-4298-82af-f3a16cb256b3","Type":"ContainerStarted","Data":"e53952d01b4942121859cb53fdfdcc23afad8ca55df1a6768a2ed49e7f244654"} Jul 17 21:49:30.657413 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:30.657389 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j" event={"ID":"1dbe15a0-0d75-4f13-98c7-5ca2510aad39","Type":"ContainerStarted","Data":"89a62119473196463cd9214a0eb22815336e4d939ff2c791c33f4ce0b92f78c5"} Jul 17 21:49:33.672079 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:33.672047 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j" event={"ID":"1dbe15a0-0d75-4f13-98c7-5ca2510aad39","Type":"ContainerStarted","Data":"60160772beb22a7a6db73e5c2b30409a0327b4527464faa90c49e09c6e22b73c"} Jul 17 21:49:33.687970 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:33.687923 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c9cc8d7dd-nj82j" podStartSLOduration=2.082328258 podStartE2EDuration="4.687907605s" totalImagesPullingTime="2.605579347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:49:33.686362487 +0000 UTC m=+210.546585988" watchObservedRunningTime="2026-07-17 21:49:33.687907605 +0000 UTC m=+210.548131109" Jul 17 21:49:45.723610 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:45.723575 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b44654799-s8xjj" event={"ID":"f66983bc-fe53-4298-82af-f3a16cb256b3","Type":"ContainerStarted","Data":"1552602c962a29f7c6e57787c1f22d5e24e7bfb1425f61a88d95fadf0d561110"} Jul 17 21:49:45.724098 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:45.723825 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b44654799-s8xjj" Jul 17 21:49:45.725480 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:45.725460 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b44654799-s8xjj" Jul 17 21:49:45.743946 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:45.743900 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b44654799-s8xjj" podStartSLOduration=1.700916192 podStartE2EDuration="16.743886035s" totalImagesPullingTime="15.042969843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:49:45.74290113 +0000 UTC m=+222.603124632" watchObservedRunningTime="2026-07-17 21:49:45.743886035 +0000 UTC m=+222.604109537" Jul 17 21:49:55.341739 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:55.341703 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rcln"] Jul 17 21:49:55.345073 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:55.345056 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rcln" Jul 17 21:49:55.347663 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:55.347640 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 17 21:49:55.347792 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:55.347640 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 17 21:49:55.347792 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:55.347642 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-w54sg\"" Jul 17 21:49:55.362129 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:55.362105 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rcln"] Jul 17 21:49:55.497144 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:55.497109 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/560d6f1d-c01d-4d91-b37b-c3f2c6656e6e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8rcln\" (UID: \"560d6f1d-c01d-4d91-b37b-c3f2c6656e6e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rcln" Jul 17 21:49:55.497337 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:55.497172 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twd2q\" (UniqueName: \"kubernetes.io/projected/560d6f1d-c01d-4d91-b37b-c3f2c6656e6e-kube-api-access-twd2q\") pod \"cert-manager-operator-controller-manager-55c444bf86-8rcln\" (UID: \"560d6f1d-c01d-4d91-b37b-c3f2c6656e6e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rcln" Jul 17 21:49:55.598099 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:55.598026 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twd2q\" (UniqueName: \"kubernetes.io/projected/560d6f1d-c01d-4d91-b37b-c3f2c6656e6e-kube-api-access-twd2q\") pod \"cert-manager-operator-controller-manager-55c444bf86-8rcln\" (UID: \"560d6f1d-c01d-4d91-b37b-c3f2c6656e6e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rcln" Jul 17 21:49:55.598099 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:55.598089 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/560d6f1d-c01d-4d91-b37b-c3f2c6656e6e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8rcln\" (UID: \"560d6f1d-c01d-4d91-b37b-c3f2c6656e6e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rcln" Jul 17 21:49:55.598400 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:55.598385 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/560d6f1d-c01d-4d91-b37b-c3f2c6656e6e-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-8rcln\" (UID: \"560d6f1d-c01d-4d91-b37b-c3f2c6656e6e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rcln" Jul 17 21:49:55.606629 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:55.606602 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twd2q\" (UniqueName: \"kubernetes.io/projected/560d6f1d-c01d-4d91-b37b-c3f2c6656e6e-kube-api-access-twd2q\") pod \"cert-manager-operator-controller-manager-55c444bf86-8rcln\" (UID: \"560d6f1d-c01d-4d91-b37b-c3f2c6656e6e\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rcln" Jul 17 21:49:55.654571 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:55.654548 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rcln" Jul 17 21:49:55.799394 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:55.799372 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rcln"] Jul 17 21:49:55.801404 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:49:55.801381 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560d6f1d_c01d_4d91_b37b_c3f2c6656e6e.slice/crio-c1727d9c7b28d8899cd4607f19a051115a5b6e7a18c00848105b020d07b085a2 WatchSource:0}: Error finding container c1727d9c7b28d8899cd4607f19a051115a5b6e7a18c00848105b020d07b085a2: Status 404 returned error can't find the container with id c1727d9c7b28d8899cd4607f19a051115a5b6e7a18c00848105b020d07b085a2 Jul 17 21:49:56.767654 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:49:56.767622 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rcln" event={"ID":"560d6f1d-c01d-4d91-b37b-c3f2c6656e6e","Type":"ContainerStarted","Data":"c1727d9c7b28d8899cd4607f19a051115a5b6e7a18c00848105b020d07b085a2"} Jul 17 21:50:08.822202 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:08.822167 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rcln" event={"ID":"560d6f1d-c01d-4d91-b37b-c3f2c6656e6e","Type":"ContainerStarted","Data":"7c668526c4e1c3b4db4a40dca414c7757ca91b0bf3cbc968653e98813f66caf4"} Jul 17 21:50:08.847968 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:08.847922 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-8rcln" podStartSLOduration=1.6822210690000001 podStartE2EDuration="13.847908031s" totalImagesPullingTime="12.165686962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:49:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:50:08.847690783 +0000 UTC m=+245.707914297" watchObservedRunningTime="2026-07-17 21:50:08.847908031 +0000 UTC m=+245.708131532" Jul 17 21:50:11.219936 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.219898 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-r4zck"] Jul 17 21:50:11.222153 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.222137 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r4zck" Jul 17 21:50:11.224574 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.224549 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 17 21:50:11.224724 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.224703 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 17 21:50:11.225416 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.225400 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vvx44\"" Jul 17 21:50:11.233859 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.233837 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-r4zck"] Jul 17 21:50:11.334256 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.334228 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq6vw\" (UniqueName: \"kubernetes.io/projected/5d8d4f5c-57d8-416c-9f44-79a4084d4d4b-kube-api-access-lq6vw\") pod \"cert-manager-cainjector-559cdd8c87-r4zck\" (UID: \"5d8d4f5c-57d8-416c-9f44-79a4084d4d4b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r4zck" Jul 17 21:50:11.334406 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.334267 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d8d4f5c-57d8-416c-9f44-79a4084d4d4b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-r4zck\" (UID: \"5d8d4f5c-57d8-416c-9f44-79a4084d4d4b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r4zck" Jul 17 21:50:11.370066 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.370039 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sn6qs"] Jul 17 21:50:11.372329 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.372313 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-sn6qs" Jul 17 21:50:11.378937 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.378918 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-9dmkk\"" Jul 17 21:50:11.412054 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.412027 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sn6qs"] Jul 17 21:50:11.434848 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.434820 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq6vw\" (UniqueName: \"kubernetes.io/projected/5d8d4f5c-57d8-416c-9f44-79a4084d4d4b-kube-api-access-lq6vw\") pod \"cert-manager-cainjector-559cdd8c87-r4zck\" (UID: \"5d8d4f5c-57d8-416c-9f44-79a4084d4d4b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r4zck" Jul 17 21:50:11.434961 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.434852 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxhdp\" (UniqueName: \"kubernetes.io/projected/2e8c799d-2d22-489d-89bb-844d919237fd-kube-api-access-lxhdp\") pod \"cert-manager-webhook-6486ffd997-sn6qs\" (UID: \"2e8c799d-2d22-489d-89bb-844d919237fd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sn6qs" Jul 17 21:50:11.434961 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.434891 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d8d4f5c-57d8-416c-9f44-79a4084d4d4b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-r4zck\" (UID: \"5d8d4f5c-57d8-416c-9f44-79a4084d4d4b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r4zck" Jul 17 21:50:11.435032 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.435001 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e8c799d-2d22-489d-89bb-844d919237fd-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sn6qs\" (UID: \"2e8c799d-2d22-489d-89bb-844d919237fd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sn6qs" Jul 17 21:50:11.452208 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.452190 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq6vw\" (UniqueName: \"kubernetes.io/projected/5d8d4f5c-57d8-416c-9f44-79a4084d4d4b-kube-api-access-lq6vw\") pod \"cert-manager-cainjector-559cdd8c87-r4zck\" (UID: \"5d8d4f5c-57d8-416c-9f44-79a4084d4d4b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r4zck" Jul 17 21:50:11.453361 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.453341 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5d8d4f5c-57d8-416c-9f44-79a4084d4d4b-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-r4zck\" (UID: \"5d8d4f5c-57d8-416c-9f44-79a4084d4d4b\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-r4zck" Jul 17 21:50:11.531386 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.531312 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r4zck" Jul 17 21:50:11.535608 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.535582 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxhdp\" (UniqueName: \"kubernetes.io/projected/2e8c799d-2d22-489d-89bb-844d919237fd-kube-api-access-lxhdp\") pod \"cert-manager-webhook-6486ffd997-sn6qs\" (UID: \"2e8c799d-2d22-489d-89bb-844d919237fd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sn6qs" Jul 17 21:50:11.535715 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.535700 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e8c799d-2d22-489d-89bb-844d919237fd-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sn6qs\" (UID: \"2e8c799d-2d22-489d-89bb-844d919237fd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sn6qs" Jul 17 21:50:11.563017 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.562988 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxhdp\" (UniqueName: \"kubernetes.io/projected/2e8c799d-2d22-489d-89bb-844d919237fd-kube-api-access-lxhdp\") pod \"cert-manager-webhook-6486ffd997-sn6qs\" (UID: \"2e8c799d-2d22-489d-89bb-844d919237fd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sn6qs" Jul 17 21:50:11.563124 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.563085 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2e8c799d-2d22-489d-89bb-844d919237fd-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sn6qs\" (UID: \"2e8c799d-2d22-489d-89bb-844d919237fd\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sn6qs" Jul 17 21:50:11.664891 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.664862 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-r4zck"] Jul 17 21:50:11.668290 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:50:11.668265 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d8d4f5c_57d8_416c_9f44_79a4084d4d4b.slice/crio-80bf2ff57eeda065c9d751258eacc40aafabd116e0d1d1b564f0d269b1fd6f71 WatchSource:0}: Error finding container 80bf2ff57eeda065c9d751258eacc40aafabd116e0d1d1b564f0d269b1fd6f71: Status 404 returned error can't find the container with id 80bf2ff57eeda065c9d751258eacc40aafabd116e0d1d1b564f0d269b1fd6f71 Jul 17 21:50:11.681037 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.681015 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-sn6qs" Jul 17 21:50:11.811302 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.811194 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sn6qs"] Jul 17 21:50:11.813363 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:50:11.813335 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e8c799d_2d22_489d_89bb_844d919237fd.slice/crio-a217578db11f4103000479d73e33a5f2522e93d7baf0857a363e0b41ab7bb134 WatchSource:0}: Error finding container a217578db11f4103000479d73e33a5f2522e93d7baf0857a363e0b41ab7bb134: Status 404 returned error can't find the container with id a217578db11f4103000479d73e33a5f2522e93d7baf0857a363e0b41ab7bb134 Jul 17 21:50:11.834470 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.834446 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-sn6qs" event={"ID":"2e8c799d-2d22-489d-89bb-844d919237fd","Type":"ContainerStarted","Data":"a217578db11f4103000479d73e33a5f2522e93d7baf0857a363e0b41ab7bb134"} Jul 17 21:50:11.835610 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:11.835587 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r4zck" event={"ID":"5d8d4f5c-57d8-416c-9f44-79a4084d4d4b","Type":"ContainerStarted","Data":"80bf2ff57eeda065c9d751258eacc40aafabd116e0d1d1b564f0d269b1fd6f71"} Jul 17 21:50:15.857549 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:15.857518 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-sn6qs" event={"ID":"2e8c799d-2d22-489d-89bb-844d919237fd","Type":"ContainerStarted","Data":"9aef8a41cf51493e4d0681e9a28f1c2ca6683abf0968e83ee63df494ef99b673"} Jul 17 21:50:15.857976 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:15.857626 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-sn6qs" Jul 17 21:50:15.859153 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:15.859124 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r4zck" event={"ID":"5d8d4f5c-57d8-416c-9f44-79a4084d4d4b","Type":"ContainerStarted","Data":"50bdd059f20998a1750d17e23fb809b2a71d5bbb668668404c1e95a01e9e0ea4"} Jul 17 21:50:15.874645 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:15.874601 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-sn6qs" podStartSLOduration=1.682910062 podStartE2EDuration="4.874587898s" totalImagesPullingTime="3.191677836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:50:15.8728205 +0000 UTC m=+252.733043999" watchObservedRunningTime="2026-07-17 21:50:15.874587898 +0000 UTC m=+252.734811382" Jul 17 21:50:15.892053 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:15.892016 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-r4zck" podStartSLOduration=1.549088636 podStartE2EDuration="4.892002706s" totalImagesPullingTime="3.34291407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:50:15.890628158 +0000 UTC m=+252.750851665" watchObservedRunningTime="2026-07-17 21:50:15.892002706 +0000 UTC m=+252.752226208" Jul 17 21:50:21.866265 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:21.866228 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-sn6qs" Jul 17 21:50:46.076577 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.076464 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc"] Jul 17 21:50:46.081924 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.081905 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" Jul 17 21:50:46.106196 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.106173 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 17 21:50:46.106196 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.106193 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 17 21:50:46.106423 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.106317 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 17 21:50:46.106423 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.106381 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 17 21:50:46.106423 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.106406 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rl47g\"" Jul 17 21:50:46.164924 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.164900 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc"] Jul 17 21:50:46.243009 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.242971 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6cde31f1-f65b-4e27-9d06-8225d0991a44-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bff9999cf-fr4dc\" (UID: \"6cde31f1-f65b-4e27-9d06-8225d0991a44\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" Jul 17 21:50:46.243191 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.243037 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cde31f1-f65b-4e27-9d06-8225d0991a44-tmp\") pod \"opendatahub-operator-controller-manager-bff9999cf-fr4dc\" (UID: \"6cde31f1-f65b-4e27-9d06-8225d0991a44\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" Jul 17 21:50:46.243191 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.243070 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6cde31f1-f65b-4e27-9d06-8225d0991a44-webhook-cert\") pod \"opendatahub-operator-controller-manager-bff9999cf-fr4dc\" (UID: \"6cde31f1-f65b-4e27-9d06-8225d0991a44\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" Jul 17 21:50:46.243191 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.243100 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6cde31f1-f65b-4e27-9d06-8225d0991a44-manifests\") pod \"opendatahub-operator-controller-manager-bff9999cf-fr4dc\" (UID: \"6cde31f1-f65b-4e27-9d06-8225d0991a44\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" Jul 17 21:50:46.243302 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.243179 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnng6\" (UniqueName: \"kubernetes.io/projected/6cde31f1-f65b-4e27-9d06-8225d0991a44-kube-api-access-rnng6\") pod \"opendatahub-operator-controller-manager-bff9999cf-fr4dc\" (UID: \"6cde31f1-f65b-4e27-9d06-8225d0991a44\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" Jul 17 21:50:46.344571 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.344469 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6cde31f1-f65b-4e27-9d06-8225d0991a44-webhook-cert\") pod \"opendatahub-operator-controller-manager-bff9999cf-fr4dc\" (UID: \"6cde31f1-f65b-4e27-9d06-8225d0991a44\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" Jul 17 21:50:46.344571 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.344521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6cde31f1-f65b-4e27-9d06-8225d0991a44-manifests\") pod \"opendatahub-operator-controller-manager-bff9999cf-fr4dc\" (UID: \"6cde31f1-f65b-4e27-9d06-8225d0991a44\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" Jul 17 21:50:46.344779 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.344607 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnng6\" (UniqueName: \"kubernetes.io/projected/6cde31f1-f65b-4e27-9d06-8225d0991a44-kube-api-access-rnng6\") pod \"opendatahub-operator-controller-manager-bff9999cf-fr4dc\" (UID: \"6cde31f1-f65b-4e27-9d06-8225d0991a44\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" Jul 17 21:50:46.344779 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.344662 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6cde31f1-f65b-4e27-9d06-8225d0991a44-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bff9999cf-fr4dc\" (UID: \"6cde31f1-f65b-4e27-9d06-8225d0991a44\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" Jul 17 21:50:46.344779 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.344709 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cde31f1-f65b-4e27-9d06-8225d0991a44-tmp\") pod \"opendatahub-operator-controller-manager-bff9999cf-fr4dc\" (UID: \"6cde31f1-f65b-4e27-9d06-8225d0991a44\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" Jul 17 21:50:46.345049 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.345026 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6cde31f1-f65b-4e27-9d06-8225d0991a44-manifests\") pod \"opendatahub-operator-controller-manager-bff9999cf-fr4dc\" (UID: \"6cde31f1-f65b-4e27-9d06-8225d0991a44\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" Jul 17 21:50:46.346924 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.346893 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6cde31f1-f65b-4e27-9d06-8225d0991a44-tmp\") pod \"opendatahub-operator-controller-manager-bff9999cf-fr4dc\" (UID: \"6cde31f1-f65b-4e27-9d06-8225d0991a44\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" Jul 17 21:50:46.347104 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.347084 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6cde31f1-f65b-4e27-9d06-8225d0991a44-webhook-cert\") pod \"opendatahub-operator-controller-manager-bff9999cf-fr4dc\" (UID: \"6cde31f1-f65b-4e27-9d06-8225d0991a44\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" Jul 17 21:50:46.347253 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.347237 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6cde31f1-f65b-4e27-9d06-8225d0991a44-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bff9999cf-fr4dc\" (UID: \"6cde31f1-f65b-4e27-9d06-8225d0991a44\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" Jul 17 21:50:46.366466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.366441 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnng6\" (UniqueName: \"kubernetes.io/projected/6cde31f1-f65b-4e27-9d06-8225d0991a44-kube-api-access-rnng6\") pod \"opendatahub-operator-controller-manager-bff9999cf-fr4dc\" (UID: \"6cde31f1-f65b-4e27-9d06-8225d0991a44\") " pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" Jul 17 21:50:46.391586 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.391559 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" Jul 17 21:50:46.543930 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.543909 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc"] Jul 17 21:50:46.545980 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:50:46.545952 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cde31f1_f65b_4e27_9d06_8225d0991a44.slice/crio-a849d46e0217093460037040d6d518a427dbdb924bba2fb19834596df9dc1abe WatchSource:0}: Error finding container a849d46e0217093460037040d6d518a427dbdb924bba2fb19834596df9dc1abe: Status 404 returned error can't find the container with id a849d46e0217093460037040d6d518a427dbdb924bba2fb19834596df9dc1abe Jul 17 21:50:46.998126 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:46.998085 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" event={"ID":"6cde31f1-f65b-4e27-9d06-8225d0991a44","Type":"ContainerStarted","Data":"a849d46e0217093460037040d6d518a427dbdb924bba2fb19834596df9dc1abe"} Jul 17 21:50:50.015149 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:50.015115 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" event={"ID":"6cde31f1-f65b-4e27-9d06-8225d0991a44","Type":"ContainerStarted","Data":"8b45bd876debad03390b7ff2a72c701dcfcaa57aed942c37225d37db9fab8e53"} Jul 17 21:50:51.020595 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:51.020562 2577 generic.go:356] "Generic (PLEG): container finished" podID="6cde31f1-f65b-4e27-9d06-8225d0991a44" containerID="8b45bd876debad03390b7ff2a72c701dcfcaa57aed942c37225d37db9fab8e53" exitCode=0 Jul 17 21:50:51.020975 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:51.020630 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" event={"ID":"6cde31f1-f65b-4e27-9d06-8225d0991a44","Type":"ContainerDied","Data":"8b45bd876debad03390b7ff2a72c701dcfcaa57aed942c37225d37db9fab8e53"} Jul 17 21:50:52.027433 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:52.027399 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" event={"ID":"6cde31f1-f65b-4e27-9d06-8225d0991a44","Type":"ContainerStarted","Data":"4855f45248b454b95ee6394f9b777503fa6f907acaa78e9b7bf2ffee55a60197"} Jul 17 21:50:52.027853 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:52.027524 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" Jul 17 21:50:52.053309 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:50:52.053253 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" podStartSLOduration=1.777557198 podStartE2EDuration="6.053236623s" totalImagesPullingTime="3.220263865s" totalInitContainerRuntime="1.05541556s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:50:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:50:52.052293533 +0000 UTC m=+288.912517086" watchObservedRunningTime="2026-07-17 21:50:52.053236623 +0000 UTC m=+288.913460124" Jul 17 21:51:03.034206 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:03.034178 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-bff9999cf-fr4dc" Jul 17 21:51:03.548358 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:03.548327 2577 kubelet.go:1648] "Image garbage collection succeeded" Jul 17 21:51:08.753747 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.753713 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9"] Jul 17 21:51:08.758080 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.758065 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.761833 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.761808 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 17 21:51:08.761946 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.761808 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 17 21:51:08.763131 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.763112 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 17 21:51:08.763265 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.763176 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 21:51:08.763265 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.763248 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 21:51:08.763398 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.763341 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 17 21:51:08.764097 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.764079 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4qrs7\"" Jul 17 21:51:08.778514 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.778467 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9"] Jul 17 21:51:08.852583 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.852544 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-cacerts\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.852783 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.852763 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.852855 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.852796 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-token\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.852919 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.852893 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.852970 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.852942 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.853051 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.853031 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb5ww\" (UniqueName: \"kubernetes.io/projected/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-kube-api-access-hb5ww\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.853166 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.853152 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-local-certs\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.954506 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.954466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-cacerts\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.954668 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.954552 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.954668 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.954581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-token\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.954668 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.954621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.954668 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.954650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.954888 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.954703 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb5ww\" (UniqueName: \"kubernetes.io/projected/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-kube-api-access-hb5ww\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.954888 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.954770 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-local-certs\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.955235 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.955201 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.957118 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.957097 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.957226 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.957165 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.957401 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.957384 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-local-certs\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.957466 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.957437 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-cacerts\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.968925 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.968898 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb5ww\" (UniqueName: \"kubernetes.io/projected/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-kube-api-access-hb5ww\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:08.971851 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:08.971833 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-token\") pod \"istiod-openshift-gateway-7574459656-lh4j9\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:09.068130 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:09.068033 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:09.222137 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:09.222112 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9"] Jul 17 21:51:09.225348 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:51:09.225306 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2edac6fc_d2ac_4a1a_b6f0_a49c15a101fc.slice/crio-ce318e5f82bc907cde7c404b74c55bd7eebaf2d70e44042dd68b0280488fded8 WatchSource:0}: Error finding container ce318e5f82bc907cde7c404b74c55bd7eebaf2d70e44042dd68b0280488fded8: Status 404 returned error can't find the container with id ce318e5f82bc907cde7c404b74c55bd7eebaf2d70e44042dd68b0280488fded8 Jul 17 21:51:09.227342 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:09.227323 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 21:51:10.115433 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:10.115390 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" event={"ID":"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc","Type":"ContainerStarted","Data":"ce318e5f82bc907cde7c404b74c55bd7eebaf2d70e44042dd68b0280488fded8"} Jul 17 21:51:11.628831 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:11.628787 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:51:11.629107 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:11.628863 2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"} Jul 17 21:51:12.134834 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:12.134792 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" event={"ID":"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc","Type":"ContainerStarted","Data":"d7780507895a06fb1ab27f43a667aeb83e98499243d0f1c7e6f25dc376b1ae56"} Jul 17 21:51:12.135306 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:12.135264 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:12.137304 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:12.137270 2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-lh4j9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 17 21:51:12.137445 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:12.137331 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" podUID="2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 17 21:51:12.157732 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:12.157683 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" podStartSLOduration=1.7566883679999998 podStartE2EDuration="4.157667807s" totalImagesPullingTime="2.400979439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:51:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:51:12.156289975 +0000 UTC m=+309.016513522" watchObservedRunningTime="2026-07-17 21:51:12.157667807 +0000 UTC m=+309.017891309" Jul 17 21:51:13.140726 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:13.140697 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:14.172284 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.172252 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt"] Jul 17 21:51:14.175686 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.175667 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt" Jul 17 21:51:14.178323 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.178304 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 17 21:51:14.178705 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.178688 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 17 21:51:14.178775 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.178703 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 17 21:51:14.178953 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.178939 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 17 21:51:14.179004 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.178950 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mq6fg\"" Jul 17 21:51:14.179004 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.178969 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 17 21:51:14.187177 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.187158 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt"] Jul 17 21:51:14.311733 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.311701 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25rq6\" (UniqueName: \"kubernetes.io/projected/34b9aba1-2c4a-4440-8a41-c33041695e06-kube-api-access-25rq6\") pod \"lws-controller-manager-68687b8cf-lcrjt\" (UID: \"34b9aba1-2c4a-4440-8a41-c33041695e06\") " pod="openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt" Jul 17 21:51:14.311925 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.311779 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34b9aba1-2c4a-4440-8a41-c33041695e06-cert\") pod \"lws-controller-manager-68687b8cf-lcrjt\" (UID: \"34b9aba1-2c4a-4440-8a41-c33041695e06\") " pod="openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt" Jul 17 21:51:14.311925 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.311827 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/34b9aba1-2c4a-4440-8a41-c33041695e06-metrics-cert\") pod \"lws-controller-manager-68687b8cf-lcrjt\" (UID: \"34b9aba1-2c4a-4440-8a41-c33041695e06\") " pod="openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt" Jul 17 21:51:14.311925 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.311922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/34b9aba1-2c4a-4440-8a41-c33041695e06-manager-config\") pod \"lws-controller-manager-68687b8cf-lcrjt\" (UID: \"34b9aba1-2c4a-4440-8a41-c33041695e06\") " pod="openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt" Jul 17 21:51:14.412460 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.412430 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25rq6\" (UniqueName: \"kubernetes.io/projected/34b9aba1-2c4a-4440-8a41-c33041695e06-kube-api-access-25rq6\") pod \"lws-controller-manager-68687b8cf-lcrjt\" (UID: \"34b9aba1-2c4a-4440-8a41-c33041695e06\") " pod="openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt" Jul 17 21:51:14.412672 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.412478 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34b9aba1-2c4a-4440-8a41-c33041695e06-cert\") pod \"lws-controller-manager-68687b8cf-lcrjt\" (UID: \"34b9aba1-2c4a-4440-8a41-c33041695e06\") " pod="openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt" Jul 17 21:51:14.412672 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.412563 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/34b9aba1-2c4a-4440-8a41-c33041695e06-metrics-cert\") pod \"lws-controller-manager-68687b8cf-lcrjt\" (UID: \"34b9aba1-2c4a-4440-8a41-c33041695e06\") " pod="openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt" Jul 17 21:51:14.412672 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.412656 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/34b9aba1-2c4a-4440-8a41-c33041695e06-manager-config\") pod \"lws-controller-manager-68687b8cf-lcrjt\" (UID: \"34b9aba1-2c4a-4440-8a41-c33041695e06\") " pod="openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt" Jul 17 21:51:14.413287 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.413264 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/34b9aba1-2c4a-4440-8a41-c33041695e06-manager-config\") pod \"lws-controller-manager-68687b8cf-lcrjt\" (UID: \"34b9aba1-2c4a-4440-8a41-c33041695e06\") " pod="openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt" Jul 17 21:51:14.415048 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.415027 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/34b9aba1-2c4a-4440-8a41-c33041695e06-metrics-cert\") pod \"lws-controller-manager-68687b8cf-lcrjt\" (UID: \"34b9aba1-2c4a-4440-8a41-c33041695e06\") " pod="openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt" Jul 17 21:51:14.415141 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.415071 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34b9aba1-2c4a-4440-8a41-c33041695e06-cert\") pod \"lws-controller-manager-68687b8cf-lcrjt\" (UID: \"34b9aba1-2c4a-4440-8a41-c33041695e06\") " pod="openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt" Jul 17 21:51:14.419948 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.419930 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25rq6\" (UniqueName: \"kubernetes.io/projected/34b9aba1-2c4a-4440-8a41-c33041695e06-kube-api-access-25rq6\") pod \"lws-controller-manager-68687b8cf-lcrjt\" (UID: \"34b9aba1-2c4a-4440-8a41-c33041695e06\") " pod="openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt" Jul 17 21:51:14.485667 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.485641 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt" Jul 17 21:51:14.626081 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:14.626057 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt"] Jul 17 21:51:14.628443 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:51:14.628416 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b9aba1_2c4a_4440_8a41_c33041695e06.slice/crio-fee65d06503750889448a5cce97122e8acaefb21a8d50d49235a5c93c49d7b43 WatchSource:0}: Error finding container fee65d06503750889448a5cce97122e8acaefb21a8d50d49235a5c93c49d7b43: Status 404 returned error can't find the container with id fee65d06503750889448a5cce97122e8acaefb21a8d50d49235a5c93c49d7b43 Jul 17 21:51:15.149756 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:15.149665 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt" event={"ID":"34b9aba1-2c4a-4440-8a41-c33041695e06","Type":"ContainerStarted","Data":"fee65d06503750889448a5cce97122e8acaefb21a8d50d49235a5c93c49d7b43"} Jul 17 21:51:18.168614 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:18.168577 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt" event={"ID":"34b9aba1-2c4a-4440-8a41-c33041695e06","Type":"ContainerStarted","Data":"ec034d54a93f20198018eb9ebd76e47bdc258309bcff8837a08249e709b43805"} Jul 17 21:51:18.169098 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:18.168723 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt" Jul 17 21:51:18.233620 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:18.233568 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt" podStartSLOduration=1.704365169 podStartE2EDuration="4.233551888s" totalImagesPullingTime="2.529186719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:51:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:51:18.232579378 +0000 UTC m=+315.092802881" watchObservedRunningTime="2026-07-17 21:51:18.233551888 +0000 UTC m=+315.093775390" Jul 17 21:51:21.706768 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:21.706731 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9"] Jul 17 21:51:21.707227 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:21.706918 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" podUID="2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc" containerName="discovery" containerID="cri-o://d7780507895a06fb1ab27f43a667aeb83e98499243d0f1c7e6f25dc376b1ae56" gracePeriod=30 Jul 17 21:51:21.949396 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:21.949370 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:21.982529 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:21.982440 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-cacerts\") pod \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " Jul 17 21:51:21.982529 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:21.982512 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hb5ww\" (UniqueName: \"kubernetes.io/projected/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-kube-api-access-hb5ww\") pod \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " Jul 17 21:51:21.982771 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:21.982542 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-token\") pod \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " Jul 17 21:51:21.982771 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:21.982582 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-csr-ca-configmap\") pod \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " Jul 17 21:51:21.982771 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:21.982601 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-local-certs\") pod \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " Jul 17 21:51:21.982771 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:21.982620 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-csr-dns-cert\") pod \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " Jul 17 21:51:21.982771 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:21.982647 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-kubeconfig\") pod \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\" (UID: \"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc\") " Jul 17 21:51:21.983186 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:21.983153 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc" (UID: "2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:51:21.985141 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:21.985112 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-local-certs" (OuterVolumeSpecName: "local-certs") pod "2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc" (UID: "2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:51:21.985141 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:21.985127 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc" (UID: "2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:51:21.985292 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:21.985161 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-cacerts" (OuterVolumeSpecName: "cacerts") pod "2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc" (UID: "2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:51:21.985292 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:21.985254 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-token" (OuterVolumeSpecName: "istio-token") pod "2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc" (UID: "2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:51:21.985363 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:21.985321 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-kube-api-access-hb5ww" (OuterVolumeSpecName: "kube-api-access-hb5ww") pod "2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc" (UID: "2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc"). InnerVolumeSpecName "kube-api-access-hb5ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:51:21.985523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:21.985503 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc" (UID: "2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:51:22.083537 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:22.083478 2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-cacerts\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:51:22.083537 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:22.083539 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb5ww\" (UniqueName: \"kubernetes.io/projected/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-kube-api-access-hb5ww\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:51:22.083729 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:22.083555 2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-token\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:51:22.083729 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:22.083565 2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-csr-ca-configmap\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:51:22.083729 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:22.083574 2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-local-certs\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:51:22.083729 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:22.083582 2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-csr-dns-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:51:22.083729 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:22.083590 2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc-istio-kubeconfig\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:51:22.191768 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:22.191737 2577 generic.go:356] "Generic (PLEG): container finished" podID="2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc" containerID="d7780507895a06fb1ab27f43a667aeb83e98499243d0f1c7e6f25dc376b1ae56" exitCode=0 Jul 17 21:51:22.191925 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:22.191781 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" event={"ID":"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc","Type":"ContainerDied","Data":"d7780507895a06fb1ab27f43a667aeb83e98499243d0f1c7e6f25dc376b1ae56"} Jul 17 21:51:22.191925 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:22.191793 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" Jul 17 21:51:22.191925 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:22.191829 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9" event={"ID":"2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc","Type":"ContainerDied","Data":"ce318e5f82bc907cde7c404b74c55bd7eebaf2d70e44042dd68b0280488fded8"} Jul 17 21:51:22.191925 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:22.191856 2577 scope.go:117] "RemoveContainer" containerID="d7780507895a06fb1ab27f43a667aeb83e98499243d0f1c7e6f25dc376b1ae56" Jul 17 21:51:22.201419 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:22.201402 2577 scope.go:117] "RemoveContainer" containerID="d7780507895a06fb1ab27f43a667aeb83e98499243d0f1c7e6f25dc376b1ae56" Jul 17 21:51:22.201687 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:51:22.201670 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7780507895a06fb1ab27f43a667aeb83e98499243d0f1c7e6f25dc376b1ae56\": container with ID starting with d7780507895a06fb1ab27f43a667aeb83e98499243d0f1c7e6f25dc376b1ae56 not found: ID does not exist" containerID="d7780507895a06fb1ab27f43a667aeb83e98499243d0f1c7e6f25dc376b1ae56" Jul 17 21:51:22.201755 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:22.201698 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7780507895a06fb1ab27f43a667aeb83e98499243d0f1c7e6f25dc376b1ae56"} err="failed to get container status \"d7780507895a06fb1ab27f43a667aeb83e98499243d0f1c7e6f25dc376b1ae56\": rpc error: code = NotFound desc = could not find container \"d7780507895a06fb1ab27f43a667aeb83e98499243d0f1c7e6f25dc376b1ae56\": container with ID starting with d7780507895a06fb1ab27f43a667aeb83e98499243d0f1c7e6f25dc376b1ae56 not found: ID does not exist" Jul 17 21:51:22.229208 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:22.229181 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9"] Jul 17 21:51:22.245514 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:22.245442 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-lh4j9"] Jul 17 21:51:23.699704 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:23.699677 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc" path="/var/lib/kubelet/pods/2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc/volumes" Jul 17 21:51:29.175378 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:51:29.175346 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-68687b8cf-lcrjt" Jul 17 21:52:07.755454 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:07.755422 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vmggk"] Jul 17 21:52:07.755861 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:07.755759 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc" containerName="discovery" Jul 17 21:52:07.755861 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:07.755770 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc" containerName="discovery" Jul 17 21:52:07.755861 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:07.755851 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2edac6fc-d2ac-4a1a-b6f0-a49c15a101fc" containerName="discovery" Jul 17 21:52:07.760144 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:07.760129 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vmggk" Jul 17 21:52:07.763669 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:07.763643 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 17 21:52:07.763811 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:07.763643 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-qvvz9\"" Jul 17 21:52:07.763811 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:07.763783 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 17 21:52:07.777536 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:07.777480 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vmggk"] Jul 17 21:52:07.867978 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:07.867948 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctqmx\" (UniqueName: \"kubernetes.io/projected/db508d6b-a37a-46e3-b7de-12b4ead995e6-kube-api-access-ctqmx\") pod \"kuadrant-operator-catalog-vmggk\" (UID: \"db508d6b-a37a-46e3-b7de-12b4ead995e6\") " pod="kuadrant-system/kuadrant-operator-catalog-vmggk" Jul 17 21:52:07.969044 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:07.969013 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctqmx\" (UniqueName: \"kubernetes.io/projected/db508d6b-a37a-46e3-b7de-12b4ead995e6-kube-api-access-ctqmx\") pod \"kuadrant-operator-catalog-vmggk\" (UID: \"db508d6b-a37a-46e3-b7de-12b4ead995e6\") " pod="kuadrant-system/kuadrant-operator-catalog-vmggk" Jul 17 21:52:07.977084 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:07.977060 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctqmx\" (UniqueName: \"kubernetes.io/projected/db508d6b-a37a-46e3-b7de-12b4ead995e6-kube-api-access-ctqmx\") pod \"kuadrant-operator-catalog-vmggk\" (UID: \"db508d6b-a37a-46e3-b7de-12b4ead995e6\") " pod="kuadrant-system/kuadrant-operator-catalog-vmggk" Jul 17 21:52:08.071872 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:08.071788 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vmggk" Jul 17 21:52:08.115917 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:08.115883 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vmggk"] Jul 17 21:52:08.204367 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:08.204347 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vmggk"] Jul 17 21:52:08.206553 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:52:08.206521 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb508d6b_a37a_46e3_b7de_12b4ead995e6.slice/crio-68eb419653bb0d8f16b3153073aa119bbf2c84d6c9c5e0107806fbdc71ac5bb7 WatchSource:0}: Error finding container 68eb419653bb0d8f16b3153073aa119bbf2c84d6c9c5e0107806fbdc71ac5bb7: Status 404 returned error can't find the container with id 68eb419653bb0d8f16b3153073aa119bbf2c84d6c9c5e0107806fbdc71ac5bb7 Jul 17 21:52:08.323089 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:08.323006 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4dsh5"] Jul 17 21:52:08.327565 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:08.327544 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4dsh5" Jul 17 21:52:08.333742 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:08.333718 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4dsh5"] Jul 17 21:52:08.416471 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:08.416439 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vmggk" event={"ID":"db508d6b-a37a-46e3-b7de-12b4ead995e6","Type":"ContainerStarted","Data":"68eb419653bb0d8f16b3153073aa119bbf2c84d6c9c5e0107806fbdc71ac5bb7"} Jul 17 21:52:08.474341 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:08.474308 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hxgb\" (UniqueName: \"kubernetes.io/projected/eb2ed825-fb18-4db4-8b35-1c3e61fb4d5b-kube-api-access-2hxgb\") pod \"kuadrant-operator-catalog-4dsh5\" (UID: \"eb2ed825-fb18-4db4-8b35-1c3e61fb4d5b\") " pod="kuadrant-system/kuadrant-operator-catalog-4dsh5" Jul 17 21:52:08.575345 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:08.575253 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hxgb\" (UniqueName: \"kubernetes.io/projected/eb2ed825-fb18-4db4-8b35-1c3e61fb4d5b-kube-api-access-2hxgb\") pod \"kuadrant-operator-catalog-4dsh5\" (UID: \"eb2ed825-fb18-4db4-8b35-1c3e61fb4d5b\") " pod="kuadrant-system/kuadrant-operator-catalog-4dsh5" Jul 17 21:52:08.593170 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:08.593140 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hxgb\" (UniqueName: \"kubernetes.io/projected/eb2ed825-fb18-4db4-8b35-1c3e61fb4d5b-kube-api-access-2hxgb\") pod \"kuadrant-operator-catalog-4dsh5\" (UID: \"eb2ed825-fb18-4db4-8b35-1c3e61fb4d5b\") " pod="kuadrant-system/kuadrant-operator-catalog-4dsh5" Jul 17 21:52:08.638546 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:08.638517 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4dsh5" Jul 17 21:52:08.790215 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:08.790188 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4dsh5"] Jul 17 21:52:08.795811 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:52:08.795787 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb2ed825_fb18_4db4_8b35_1c3e61fb4d5b.slice/crio-897f6c1742ae6135582b0c40d9d93fe5ea6ea48333331edc51bdd6229fafe9fa WatchSource:0}: Error finding container 897f6c1742ae6135582b0c40d9d93fe5ea6ea48333331edc51bdd6229fafe9fa: Status 404 returned error can't find the container with id 897f6c1742ae6135582b0c40d9d93fe5ea6ea48333331edc51bdd6229fafe9fa Jul 17 21:52:09.423834 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:09.423801 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4dsh5" event={"ID":"eb2ed825-fb18-4db4-8b35-1c3e61fb4d5b","Type":"ContainerStarted","Data":"897f6c1742ae6135582b0c40d9d93fe5ea6ea48333331edc51bdd6229fafe9fa"} Jul 17 21:52:10.435821 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:10.435778 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4dsh5" event={"ID":"eb2ed825-fb18-4db4-8b35-1c3e61fb4d5b","Type":"ContainerStarted","Data":"59fa279f8ed74fbe8fb036db30f4f6cbd6fed18ad8be015d0961b0053d07e6fa"} Jul 17 21:52:10.437424 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:10.437393 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vmggk" event={"ID":"db508d6b-a37a-46e3-b7de-12b4ead995e6","Type":"ContainerStarted","Data":"00eb6bcd59e7a0151d03403b208db65ae9358dbe10c6e8213f7c700f2c504761"} Jul 17 21:52:10.437571 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:10.437525 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-vmggk" podUID="db508d6b-a37a-46e3-b7de-12b4ead995e6" containerName="registry-server" containerID="cri-o://00eb6bcd59e7a0151d03403b208db65ae9358dbe10c6e8213f7c700f2c504761" gracePeriod=2 Jul 17 21:52:10.453878 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:10.453821 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-4dsh5" podStartSLOduration=1.018834323 podStartE2EDuration="2.453800019s" totalImagesPullingTime="1.434965696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:52:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:52:10.451583822 +0000 UTC m=+367.311807344" watchObservedRunningTime="2026-07-17 21:52:10.453800019 +0000 UTC m=+367.314023523" Jul 17 21:52:10.468554 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:10.468469 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-vmggk" podStartSLOduration=1.447923875 podStartE2EDuration="3.468454313s" totalImagesPullingTime="2.020530438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:52:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:52:10.468317138 +0000 UTC m=+367.328540653" watchObservedRunningTime="2026-07-17 21:52:10.468454313 +0000 UTC m=+367.328677814" Jul 17 21:52:10.694799 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:10.694768 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vmggk" Jul 17 21:52:10.797472 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:10.797387 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctqmx\" (UniqueName: \"kubernetes.io/projected/db508d6b-a37a-46e3-b7de-12b4ead995e6-kube-api-access-ctqmx\") pod \"db508d6b-a37a-46e3-b7de-12b4ead995e6\" (UID: \"db508d6b-a37a-46e3-b7de-12b4ead995e6\") " Jul 17 21:52:10.799830 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:10.799802 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db508d6b-a37a-46e3-b7de-12b4ead995e6-kube-api-access-ctqmx" (OuterVolumeSpecName: "kube-api-access-ctqmx") pod "db508d6b-a37a-46e3-b7de-12b4ead995e6" (UID: "db508d6b-a37a-46e3-b7de-12b4ead995e6"). InnerVolumeSpecName "kube-api-access-ctqmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:52:10.898858 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:10.898822 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctqmx\" (UniqueName: \"kubernetes.io/projected/db508d6b-a37a-46e3-b7de-12b4ead995e6-kube-api-access-ctqmx\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:52:11.443672 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:11.443642 2577 generic.go:356] "Generic (PLEG): container finished" podID="db508d6b-a37a-46e3-b7de-12b4ead995e6" containerID="00eb6bcd59e7a0151d03403b208db65ae9358dbe10c6e8213f7c700f2c504761" exitCode=0 Jul 17 21:52:11.444059 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:11.443684 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-vmggk" Jul 17 21:52:11.444059 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:11.443720 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vmggk" event={"ID":"db508d6b-a37a-46e3-b7de-12b4ead995e6","Type":"ContainerDied","Data":"00eb6bcd59e7a0151d03403b208db65ae9358dbe10c6e8213f7c700f2c504761"} Jul 17 21:52:11.444059 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:11.443766 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-vmggk" event={"ID":"db508d6b-a37a-46e3-b7de-12b4ead995e6","Type":"ContainerDied","Data":"68eb419653bb0d8f16b3153073aa119bbf2c84d6c9c5e0107806fbdc71ac5bb7"} Jul 17 21:52:11.444059 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:11.443786 2577 scope.go:117] "RemoveContainer" containerID="00eb6bcd59e7a0151d03403b208db65ae9358dbe10c6e8213f7c700f2c504761" Jul 17 21:52:11.453612 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:11.453596 2577 scope.go:117] "RemoveContainer" containerID="00eb6bcd59e7a0151d03403b208db65ae9358dbe10c6e8213f7c700f2c504761" Jul 17 21:52:11.453888 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:52:11.453869 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00eb6bcd59e7a0151d03403b208db65ae9358dbe10c6e8213f7c700f2c504761\": container with ID starting with 00eb6bcd59e7a0151d03403b208db65ae9358dbe10c6e8213f7c700f2c504761 not found: ID does not exist" containerID="00eb6bcd59e7a0151d03403b208db65ae9358dbe10c6e8213f7c700f2c504761" Jul 17 21:52:11.453923 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:11.453897 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00eb6bcd59e7a0151d03403b208db65ae9358dbe10c6e8213f7c700f2c504761"} err="failed to get container status \"00eb6bcd59e7a0151d03403b208db65ae9358dbe10c6e8213f7c700f2c504761\": rpc error: code = NotFound desc = could not find container \"00eb6bcd59e7a0151d03403b208db65ae9358dbe10c6e8213f7c700f2c504761\": container with ID starting with 00eb6bcd59e7a0151d03403b208db65ae9358dbe10c6e8213f7c700f2c504761 not found: ID does not exist" Jul 17 21:52:11.464403 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:11.464377 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vmggk"] Jul 17 21:52:11.468539 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:11.468518 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-vmggk"] Jul 17 21:52:11.700456 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:11.700429 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db508d6b-a37a-46e3-b7de-12b4ead995e6" path="/var/lib/kubelet/pods/db508d6b-a37a-46e3-b7de-12b4ead995e6/volumes" Jul 17 21:52:18.639061 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:18.639028 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-4dsh5" Jul 17 21:52:18.639547 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:18.639081 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-4dsh5" Jul 17 21:52:18.662368 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:18.662343 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-4dsh5" Jul 17 21:52:19.504173 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:19.504146 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-4dsh5" Jul 17 21:52:47.431628 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:47.431601 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-md9pb"] Jul 17 21:52:47.432127 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:47.431941 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db508d6b-a37a-46e3-b7de-12b4ead995e6" containerName="registry-server" Jul 17 21:52:47.432127 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:47.431950 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="db508d6b-a37a-46e3-b7de-12b4ead995e6" containerName="registry-server" Jul 17 21:52:47.432127 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:47.432019 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="db508d6b-a37a-46e3-b7de-12b4ead995e6" containerName="registry-server" Jul 17 21:52:47.437809 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:47.437786 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-md9pb" Jul 17 21:52:47.440089 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:47.440066 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 17 21:52:47.440209 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:47.440152 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-sx9sq\"" Jul 17 21:52:47.448825 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:47.448787 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-md9pb"] Jul 17 21:52:47.528667 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:47.528639 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m87bv\" (UniqueName: \"kubernetes.io/projected/35625c7d-b3f1-44c3-8930-0b2eb0ca5049-kube-api-access-m87bv\") pod \"dns-operator-controller-manager-864dd78c4c-md9pb\" (UID: \"35625c7d-b3f1-44c3-8930-0b2eb0ca5049\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-md9pb" Jul 17 21:52:47.629319 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:47.629294 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m87bv\" (UniqueName: \"kubernetes.io/projected/35625c7d-b3f1-44c3-8930-0b2eb0ca5049-kube-api-access-m87bv\") pod \"dns-operator-controller-manager-864dd78c4c-md9pb\" (UID: \"35625c7d-b3f1-44c3-8930-0b2eb0ca5049\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-md9pb" Jul 17 21:52:47.638053 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:47.638024 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m87bv\" (UniqueName: \"kubernetes.io/projected/35625c7d-b3f1-44c3-8930-0b2eb0ca5049-kube-api-access-m87bv\") pod \"dns-operator-controller-manager-864dd78c4c-md9pb\" (UID: \"35625c7d-b3f1-44c3-8930-0b2eb0ca5049\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-md9pb" Jul 17 21:52:47.748604 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:47.748577 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-md9pb" Jul 17 21:52:47.885307 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:47.885261 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-md9pb"] Jul 17 21:52:47.888121 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:52:47.888093 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35625c7d_b3f1_44c3_8930_0b2eb0ca5049.slice/crio-66102e083f12edaa080c9fd53f2f8d8538c2cb2eeaf61deabf631a21e3d9c366 WatchSource:0}: Error finding container 66102e083f12edaa080c9fd53f2f8d8538c2cb2eeaf61deabf631a21e3d9c366: Status 404 returned error can't find the container with id 66102e083f12edaa080c9fd53f2f8d8538c2cb2eeaf61deabf631a21e3d9c366 Jul 17 21:52:48.619315 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:48.619283 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-md9pb" event={"ID":"35625c7d-b3f1-44c3-8930-0b2eb0ca5049","Type":"ContainerStarted","Data":"66102e083f12edaa080c9fd53f2f8d8538c2cb2eeaf61deabf631a21e3d9c366"} Jul 17 21:52:50.630582 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:50.630547 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-md9pb" event={"ID":"35625c7d-b3f1-44c3-8930-0b2eb0ca5049","Type":"ContainerStarted","Data":"d496d7ffd18c9242b9690450db54aef20238d190a1d349147b240a1ce5cb7fe5"} Jul 17 21:52:50.630964 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:50.630635 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-md9pb" Jul 17 21:52:50.646951 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:50.646897 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-md9pb" podStartSLOduration=1.5635431469999999 podStartE2EDuration="3.646882516s" totalImagesPullingTime="2.083339369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:52:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:52:50.645772202 +0000 UTC m=+407.505995703" watchObservedRunningTime="2026-07-17 21:52:50.646882516 +0000 UTC m=+407.507106017" Jul 17 21:52:51.726801 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:51.726769 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-xz9mp"] Jul 17 21:52:51.730731 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:51.730710 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xz9mp" Jul 17 21:52:51.733359 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:51.733339 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-std6f\"" Jul 17 21:52:51.750557 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:51.750538 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-xz9mp"] Jul 17 21:52:51.870178 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:51.870144 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdwcs\" (UniqueName: \"kubernetes.io/projected/bd9a69c9-2654-44b4-81b5-642ec7af3834-kube-api-access-vdwcs\") pod \"authorino-operator-75cc8d5cb7-xz9mp\" (UID: \"bd9a69c9-2654-44b4-81b5-642ec7af3834\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-xz9mp" Jul 17 21:52:51.971103 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:51.971073 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdwcs\" (UniqueName: \"kubernetes.io/projected/bd9a69c9-2654-44b4-81b5-642ec7af3834-kube-api-access-vdwcs\") pod \"authorino-operator-75cc8d5cb7-xz9mp\" (UID: \"bd9a69c9-2654-44b4-81b5-642ec7af3834\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-xz9mp" Jul 17 21:52:51.979863 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:51.979804 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdwcs\" (UniqueName: \"kubernetes.io/projected/bd9a69c9-2654-44b4-81b5-642ec7af3834-kube-api-access-vdwcs\") pod \"authorino-operator-75cc8d5cb7-xz9mp\" (UID: \"bd9a69c9-2654-44b4-81b5-642ec7af3834\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-xz9mp" Jul 17 21:52:52.040802 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:52.040766 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xz9mp" Jul 17 21:52:52.178783 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:52.178761 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-xz9mp"] Jul 17 21:52:52.181272 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:52:52.181245 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9a69c9_2654_44b4_81b5_642ec7af3834.slice/crio-8f1bcf7bf56508c09e9af7a79e7aed12f33d07450478c01d165eb3b5ecec9d6f WatchSource:0}: Error finding container 8f1bcf7bf56508c09e9af7a79e7aed12f33d07450478c01d165eb3b5ecec9d6f: Status 404 returned error can't find the container with id 8f1bcf7bf56508c09e9af7a79e7aed12f33d07450478c01d165eb3b5ecec9d6f Jul 17 21:52:52.642259 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:52.642230 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xz9mp" event={"ID":"bd9a69c9-2654-44b4-81b5-642ec7af3834","Type":"ContainerStarted","Data":"8f1bcf7bf56508c09e9af7a79e7aed12f33d07450478c01d165eb3b5ecec9d6f"} Jul 17 21:52:54.654184 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:54.654104 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xz9mp" event={"ID":"bd9a69c9-2654-44b4-81b5-642ec7af3834","Type":"ContainerStarted","Data":"f20803bef24752c9e26a24d8873ac01ccd4003dde3f49c68b2ad088096d4a8b5"} Jul 17 21:52:54.654184 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:54.654176 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xz9mp" Jul 17 21:52:54.673332 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:54.673288 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xz9mp" podStartSLOduration=1.55401089 podStartE2EDuration="3.67327308s" totalImagesPullingTime="2.11926219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:52:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:52:54.672301874 +0000 UTC m=+411.532525377" watchObservedRunningTime="2026-07-17 21:52:54.67327308 +0000 UTC m=+411.533496581" Jul 17 21:52:56.691213 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:56.691175 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll"] Jul 17 21:52:56.694819 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:56.694798 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" Jul 17 21:52:56.696944 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:56.696922 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-t2xkx\"" Jul 17 21:52:56.706817 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:56.706798 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll"] Jul 17 21:52:56.713190 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:56.713167 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h95ll\" (UID: \"7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" Jul 17 21:52:56.713382 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:56.713358 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfhm8\" (UniqueName: \"kubernetes.io/projected/7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd-kube-api-access-nfhm8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h95ll\" (UID: \"7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" Jul 17 21:52:56.814206 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:56.814177 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfhm8\" (UniqueName: \"kubernetes.io/projected/7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd-kube-api-access-nfhm8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h95ll\" (UID: \"7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" Jul 17 21:52:56.814361 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:56.814232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h95ll\" (UID: \"7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" Jul 17 21:52:56.814742 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:56.814722 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h95ll\" (UID: \"7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" Jul 17 21:52:56.823167 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:56.823140 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfhm8\" (UniqueName: \"kubernetes.io/projected/7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd-kube-api-access-nfhm8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-h95ll\" (UID: \"7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" Jul 17 21:52:57.005249 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:57.005169 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" Jul 17 21:52:57.149407 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:57.149381 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll"] Jul 17 21:52:57.151729 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:52:57.151697 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd3b74e_e964_4c1a_b0a6_6af03fb2e7fd.slice/crio-88388bd5ec612478e38071314f576849bf4fffe40ef34aad5b8c41ff9440c722 WatchSource:0}: Error finding container 88388bd5ec612478e38071314f576849bf4fffe40ef34aad5b8c41ff9440c722: Status 404 returned error can't find the container with id 88388bd5ec612478e38071314f576849bf4fffe40ef34aad5b8c41ff9440c722 Jul 17 21:52:57.669632 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:52:57.669598 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" event={"ID":"7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd","Type":"ContainerStarted","Data":"88388bd5ec612478e38071314f576849bf4fffe40ef34aad5b8c41ff9440c722"} Jul 17 21:53:01.638415 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:01.638383 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-md9pb" Jul 17 21:53:02.695523 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:02.695480 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" event={"ID":"7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd","Type":"ContainerStarted","Data":"c03b2081358e8cf7e6383b1912c4b1f5110941a7fbc158b7468fc0cfcaeb2013"} Jul 17 21:53:02.695909 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:02.695566 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" Jul 17 21:53:02.718230 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:02.718178 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" podStartSLOduration=1.6588950310000001 podStartE2EDuration="6.718161843s" totalImagesPullingTime="5.059266812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:02.715719127 +0000 UTC m=+419.575942628" watchObservedRunningTime="2026-07-17 21:53:02.718161843 +0000 UTC m=+419.578385402" Jul 17 21:53:04.927829 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:04.927795 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-j4lgz"] Jul 17 21:53:04.933230 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:04.933194 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-j4lgz" Jul 17 21:53:04.935361 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:04.935337 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 17 21:53:04.935455 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:04.935370 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 17 21:53:04.935455 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:04.935388 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4wlrl\"" Jul 17 21:53:04.938538 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:04.938519 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-j4lgz"] Jul 17 21:53:04.992177 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:04.992146 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrbs6\" (UniqueName: \"kubernetes.io/projected/7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c-kube-api-access-jrbs6\") pod \"kuadrant-console-plugin-66b5774985-j4lgz\" (UID: \"7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-j4lgz" Jul 17 21:53:04.992333 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:04.992191 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-j4lgz\" (UID: \"7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-j4lgz" Jul 17 21:53:04.992333 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:04.992210 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-j4lgz\" (UID: \"7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-j4lgz" Jul 17 21:53:05.093244 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:05.093212 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-j4lgz\" (UID: \"7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-j4lgz" Jul 17 21:53:05.093244 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:05.093244 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-j4lgz\" (UID: \"7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-j4lgz" Jul 17 21:53:05.093431 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:05.093309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrbs6\" (UniqueName: \"kubernetes.io/projected/7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c-kube-api-access-jrbs6\") pod \"kuadrant-console-plugin-66b5774985-j4lgz\" (UID: \"7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-j4lgz" Jul 17 21:53:05.093431 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:53:05.093358 2577 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 17 21:53:05.093431 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:53:05.093430 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c-plugin-serving-cert podName:7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c nodeName:}" failed. No retries permitted until 2026-07-17 21:53:05.593412028 +0000 UTC m=+422.453635513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-j4lgz" (UID: "7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c") : secret "plugin-serving-cert" not found Jul 17 21:53:05.093901 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:05.093883 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-j4lgz\" (UID: \"7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-j4lgz" Jul 17 21:53:05.103613 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:05.103591 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrbs6\" (UniqueName: \"kubernetes.io/projected/7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c-kube-api-access-jrbs6\") pod \"kuadrant-console-plugin-66b5774985-j4lgz\" (UID: \"7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-j4lgz" Jul 17 21:53:05.598327 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:05.598295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-j4lgz\" (UID: \"7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-j4lgz" Jul 17 21:53:05.600954 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:05.600930 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-j4lgz\" (UID: \"7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-j4lgz" Jul 17 21:53:05.660637 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:05.660606 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-xz9mp" Jul 17 21:53:05.843689 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:05.843659 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-j4lgz" Jul 17 21:53:05.978003 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:05.977952 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-j4lgz"] Jul 17 21:53:05.980199 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:53:05.980167 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7be84fcb_5e0a_4968_8b8d_1eb9ea1a134c.slice/crio-e4af165e46669ba8fa60d3b0f47aebc9b72e49f3f147d8e50727a0afeb67bf28 WatchSource:0}: Error finding container e4af165e46669ba8fa60d3b0f47aebc9b72e49f3f147d8e50727a0afeb67bf28: Status 404 returned error can't find the container with id e4af165e46669ba8fa60d3b0f47aebc9b72e49f3f147d8e50727a0afeb67bf28 Jul 17 21:53:06.717680 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:06.717648 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-j4lgz" event={"ID":"7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c","Type":"ContainerStarted","Data":"e4af165e46669ba8fa60d3b0f47aebc9b72e49f3f147d8e50727a0afeb67bf28"} Jul 17 21:53:13.703607 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:13.703579 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" Jul 17 21:53:14.710132 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:14.710094 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj"] Jul 17 21:53:14.723643 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:14.723609 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj" Jul 17 21:53:14.731184 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:14.731123 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj"] Jul 17 21:53:14.793958 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:14.793922 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhp29\" (UniqueName: \"kubernetes.io/projected/01d62f7c-674b-4a61-a147-6ff35adc199a-kube-api-access-fhp29\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cb2zj\" (UID: \"01d62f7c-674b-4a61-a147-6ff35adc199a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj" Jul 17 21:53:14.794127 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:14.793979 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01d62f7c-674b-4a61-a147-6ff35adc199a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cb2zj\" (UID: \"01d62f7c-674b-4a61-a147-6ff35adc199a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj" Jul 17 21:53:14.895188 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:14.895153 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhp29\" (UniqueName: \"kubernetes.io/projected/01d62f7c-674b-4a61-a147-6ff35adc199a-kube-api-access-fhp29\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cb2zj\" (UID: \"01d62f7c-674b-4a61-a147-6ff35adc199a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj" Jul 17 21:53:14.895356 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:14.895196 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01d62f7c-674b-4a61-a147-6ff35adc199a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cb2zj\" (UID: \"01d62f7c-674b-4a61-a147-6ff35adc199a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj" Jul 17 21:53:14.895691 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:14.895667 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01d62f7c-674b-4a61-a147-6ff35adc199a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cb2zj\" (UID: \"01d62f7c-674b-4a61-a147-6ff35adc199a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj" Jul 17 21:53:14.924209 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:14.924179 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhp29\" (UniqueName: \"kubernetes.io/projected/01d62f7c-674b-4a61-a147-6ff35adc199a-kube-api-access-fhp29\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-cb2zj\" (UID: \"01d62f7c-674b-4a61-a147-6ff35adc199a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj" Jul 17 21:53:15.038311 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.038230 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj" Jul 17 21:53:15.474272 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.474237 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj"] Jul 17 21:53:15.477925 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.477887 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj"] Jul 17 21:53:15.488045 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.488017 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll"] Jul 17 21:53:15.488271 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.488247 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" podUID="7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd" containerName="manager" containerID="cri-o://c03b2081358e8cf7e6383b1912c4b1f5110941a7fbc158b7468fc0cfcaeb2013" gracePeriod=2 Jul 17 21:53:15.500367 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.497758 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc"] Jul 17 21:53:15.507335 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:53:15.507292 2577 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-h95ll\": pods \"kuadrant-operator-controller-manager-8684c9ff49-h95ll\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" Jul 17 21:53:15.508913 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.508863 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll"] Jul 17 21:53:15.509051 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.508990 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc" Jul 17 21:53:15.511175 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:53:15.511142 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-h95ll\" is forbidden: User \"system:node:ip-10-0-131-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-14.ec2.internal' and this object" podUID="7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" Jul 17 21:53:15.515783 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.515762 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr"] Jul 17 21:53:15.516227 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.516209 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd" containerName="manager" Jul 17 21:53:15.516227 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.516230 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd" containerName="manager" Jul 17 21:53:15.516368 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.516347 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd" containerName="manager" Jul 17 21:53:15.522712 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.522694 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr" Jul 17 21:53:15.523333 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.523302 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc"] Jul 17 21:53:15.535557 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.534950 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr"] Jul 17 21:53:15.556268 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:53:15.556217 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-h95ll\" is forbidden: User \"system:node:ip-10-0-131-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-14.ec2.internal' and this object" podUID="7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" Jul 17 21:53:15.602177 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.602143 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhvdp\" (UniqueName: \"kubernetes.io/projected/58d3303d-4dc0-4308-ba8c-b21d35c07408-kube-api-access-rhvdp\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rwmkc\" (UID: \"58d3303d-4dc0-4308-ba8c-b21d35c07408\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc" Jul 17 21:53:15.602177 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.602183 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ccfa46a-90c8-4b49-998b-1dbc8eb498e9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qcfdr\" (UID: \"7ccfa46a-90c8-4b49-998b-1dbc8eb498e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr" Jul 17 21:53:15.602389 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.602227 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/58d3303d-4dc0-4308-ba8c-b21d35c07408-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rwmkc\" (UID: \"58d3303d-4dc0-4308-ba8c-b21d35c07408\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc" Jul 17 21:53:15.602389 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.602330 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t25hx\" (UniqueName: \"kubernetes.io/projected/7ccfa46a-90c8-4b49-998b-1dbc8eb498e9-kube-api-access-t25hx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qcfdr\" (UID: \"7ccfa46a-90c8-4b49-998b-1dbc8eb498e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr" Jul 17 21:53:15.703428 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.703386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/58d3303d-4dc0-4308-ba8c-b21d35c07408-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rwmkc\" (UID: \"58d3303d-4dc0-4308-ba8c-b21d35c07408\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc" Jul 17 21:53:15.703641 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.703459 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t25hx\" (UniqueName: \"kubernetes.io/projected/7ccfa46a-90c8-4b49-998b-1dbc8eb498e9-kube-api-access-t25hx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qcfdr\" (UID: \"7ccfa46a-90c8-4b49-998b-1dbc8eb498e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr" Jul 17 21:53:15.703641 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.703575 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhvdp\" (UniqueName: \"kubernetes.io/projected/58d3303d-4dc0-4308-ba8c-b21d35c07408-kube-api-access-rhvdp\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rwmkc\" (UID: \"58d3303d-4dc0-4308-ba8c-b21d35c07408\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc" Jul 17 21:53:15.703641 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.703607 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ccfa46a-90c8-4b49-998b-1dbc8eb498e9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qcfdr\" (UID: \"7ccfa46a-90c8-4b49-998b-1dbc8eb498e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr" Jul 17 21:53:15.703817 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.703794 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/58d3303d-4dc0-4308-ba8c-b21d35c07408-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rwmkc\" (UID: \"58d3303d-4dc0-4308-ba8c-b21d35c07408\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc" Jul 17 21:53:15.703925 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.703909 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ccfa46a-90c8-4b49-998b-1dbc8eb498e9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qcfdr\" (UID: \"7ccfa46a-90c8-4b49-998b-1dbc8eb498e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr" Jul 17 21:53:15.714819 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.714789 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhvdp\" (UniqueName: \"kubernetes.io/projected/58d3303d-4dc0-4308-ba8c-b21d35c07408-kube-api-access-rhvdp\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rwmkc\" (UID: \"58d3303d-4dc0-4308-ba8c-b21d35c07408\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc" Jul 17 21:53:15.715178 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.714864 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t25hx\" (UniqueName: \"kubernetes.io/projected/7ccfa46a-90c8-4b49-998b-1dbc8eb498e9-kube-api-access-t25hx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qcfdr\" (UID: \"7ccfa46a-90c8-4b49-998b-1dbc8eb498e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr" Jul 17 21:53:15.869085 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.869001 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc" Jul 17 21:53:15.875177 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:15.875143 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr" Jul 17 21:53:28.058574 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.058543 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" Jul 17 21:53:28.060683 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:53:28.060637 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-h95ll\" is forbidden: User \"system:node:ip-10-0-131-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-14.ec2.internal' and this object" podUID="7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" Jul 17 21:53:28.061070 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.061048 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr"] Jul 17 21:53:28.063843 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:53:28.063817 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ccfa46a_90c8_4b49_998b_1dbc8eb498e9.slice/crio-7f438c86b17d367c532d170a92c5750a74656d6e657922e09e40080e27d0f2a4 WatchSource:0}: Error finding container 7f438c86b17d367c532d170a92c5750a74656d6e657922e09e40080e27d0f2a4: Status 404 returned error can't find the container with id 7f438c86b17d367c532d170a92c5750a74656d6e657922e09e40080e27d0f2a4 Jul 17 21:53:28.083962 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.083863 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc"] Jul 17 21:53:28.086650 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:53:28.086624 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58d3303d_4dc0_4308_ba8c_b21d35c07408.slice/crio-4521f87bdf92c9522912e311f8df3c70c7efcd16452e0f71dbdf3bf3cec8792b WatchSource:0}: Error finding container 4521f87bdf92c9522912e311f8df3c70c7efcd16452e0f71dbdf3bf3cec8792b: Status 404 returned error can't find the container with id 4521f87bdf92c9522912e311f8df3c70c7efcd16452e0f71dbdf3bf3cec8792b Jul 17 21:53:28.127711 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.127686 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfhm8\" (UniqueName: \"kubernetes.io/projected/7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd-kube-api-access-nfhm8\") pod \"7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd\" (UID: \"7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd\") " Jul 17 21:53:28.127829 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.127774 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd-extensions-socket-volume\") pod \"7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd\" (UID: \"7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd\") " Jul 17 21:53:28.128182 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.128151 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd" (UID: "7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:53:28.129878 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.129857 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd-kube-api-access-nfhm8" (OuterVolumeSpecName: "kube-api-access-nfhm8") pod "7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd" (UID: "7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd"). InnerVolumeSpecName "kube-api-access-nfhm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:53:28.229044 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.229008 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd-extensions-socket-volume\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:53:28.229044 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.229037 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfhm8\" (UniqueName: \"kubernetes.io/projected/7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd-kube-api-access-nfhm8\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:53:28.850633 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.850597 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-j4lgz" event={"ID":"7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c","Type":"ContainerStarted","Data":"7582dd9f379c3dc943992001a4515ec3a141fca0847a534613034031b1f87cff"} Jul 17 21:53:28.852796 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:53:28.852756 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-h95ll\" is forbidden: User \"system:node:ip-10-0-131-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-14.ec2.internal' and this object" podUID="7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" Jul 17 21:53:28.858526 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.858452 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc" event={"ID":"58d3303d-4dc0-4308-ba8c-b21d35c07408","Type":"ContainerStarted","Data":"99c8d2bd30846492590d0cab0a6784b5d32403a41ff3f57cdf31251d4a54739c"} Jul 17 21:53:28.858748 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.858556 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc" event={"ID":"58d3303d-4dc0-4308-ba8c-b21d35c07408","Type":"ContainerStarted","Data":"4521f87bdf92c9522912e311f8df3c70c7efcd16452e0f71dbdf3bf3cec8792b"} Jul 17 21:53:28.858748 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.858588 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc" Jul 17 21:53:28.860100 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.860080 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr" event={"ID":"7ccfa46a-90c8-4b49-998b-1dbc8eb498e9","Type":"ContainerStarted","Data":"d58b834ebb5ad06d5c1f78b1ad1debe7770fb6093ff19f421ea4e4ae064de839"} Jul 17 21:53:28.860200 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.860115 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr" event={"ID":"7ccfa46a-90c8-4b49-998b-1dbc8eb498e9","Type":"ContainerStarted","Data":"7f438c86b17d367c532d170a92c5750a74656d6e657922e09e40080e27d0f2a4"} Jul 17 21:53:28.860200 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.860136 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr" Jul 17 21:53:28.861340 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.861318 2577 generic.go:356] "Generic (PLEG): container finished" podID="7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd" containerID="c03b2081358e8cf7e6383b1912c4b1f5110941a7fbc158b7468fc0cfcaeb2013" exitCode=0 Jul 17 21:53:28.861423 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.861361 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" Jul 17 21:53:28.861423 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.861375 2577 scope.go:117] "RemoveContainer" containerID="c03b2081358e8cf7e6383b1912c4b1f5110941a7fbc158b7468fc0cfcaeb2013" Jul 17 21:53:28.870737 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.870718 2577 scope.go:117] "RemoveContainer" containerID="c03b2081358e8cf7e6383b1912c4b1f5110941a7fbc158b7468fc0cfcaeb2013" Jul 17 21:53:28.871146 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:53:28.871120 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c03b2081358e8cf7e6383b1912c4b1f5110941a7fbc158b7468fc0cfcaeb2013\": container with ID starting with c03b2081358e8cf7e6383b1912c4b1f5110941a7fbc158b7468fc0cfcaeb2013 not found: ID does not exist" containerID="c03b2081358e8cf7e6383b1912c4b1f5110941a7fbc158b7468fc0cfcaeb2013" Jul 17 21:53:28.871301 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.871272 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c03b2081358e8cf7e6383b1912c4b1f5110941a7fbc158b7468fc0cfcaeb2013"} err="failed to get container status \"c03b2081358e8cf7e6383b1912c4b1f5110941a7fbc158b7468fc0cfcaeb2013\": rpc error: code = NotFound desc = could not find container \"c03b2081358e8cf7e6383b1912c4b1f5110941a7fbc158b7468fc0cfcaeb2013\": container with ID starting with c03b2081358e8cf7e6383b1912c4b1f5110941a7fbc158b7468fc0cfcaeb2013 not found: ID does not exist" Jul 17 21:53:28.872272 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.872234 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-j4lgz" podStartSLOduration=2.860816356 podStartE2EDuration="24.87222084s" totalImagesPullingTime="22.011404484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:28.870969259 +0000 UTC m=+445.731192762" watchObservedRunningTime="2026-07-17 21:53:28.87222084 +0000 UTC m=+445.732444342" Jul 17 21:53:28.873030 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:53:28.873006 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-h95ll\" is forbidden: User \"system:node:ip-10-0-131-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-14.ec2.internal' and this object" podUID="7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-h95ll" Jul 17 21:53:28.893525 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.893091 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc" podStartSLOduration=13.893072194 podStartE2EDuration="13.893072194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:28.892456127 +0000 UTC m=+445.752679629" watchObservedRunningTime="2026-07-17 21:53:28.893072194 +0000 UTC m=+445.753295697" Jul 17 21:53:28.914252 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:28.914213 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr" podStartSLOduration=13.914198737 podStartE2EDuration="13.914198737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:28.912715496 +0000 UTC m=+445.772938999" watchObservedRunningTime="2026-07-17 21:53:28.914198737 +0000 UTC m=+445.774422240" Jul 17 21:53:29.701134 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:29.701099 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd" path="/var/lib/kubelet/pods/7fd3b74e-e964-4c1a-b0a6-6af03fb2e7fd/volumes" Jul 17 21:53:30.435064 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:53:30.435011 2577 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-cb2zj_kuadrant-system_01d62f7c-674b-4a61-a147-6ff35adc199a_0(11710ec3bd857a10455ac145952a7a3371f8d4024f91dab8f1c9bf487f4c017d): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-cb2zj to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 17 21:53:30.435188 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:53:30.435089 2577 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-6fb7879bc9-cb2zj_kuadrant-system_01d62f7c-674b-4a61-a147-6ff35adc199a_0(11710ec3bd857a10455ac145952a7a3371f8d4024f91dab8f1c9bf487f4c017d): error adding pod kuadrant-system_kuadrant-operator-controller-manager-6fb7879bc9-cb2zj to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj" Jul 17 21:53:30.871247 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:30.871212 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj" Jul 17 21:53:30.873485 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:53:30.873452 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cb2zj\" is forbidden: User \"system:node:ip-10-0-131-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-14.ec2.internal' and this object" podUID="01d62f7c-674b-4a61-a147-6ff35adc199a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj" Jul 17 21:53:30.880708 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:30.880690 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj" Jul 17 21:53:30.882805 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:53:30.882784 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cb2zj\" is forbidden: User \"system:node:ip-10-0-131-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-14.ec2.internal' and this object" podUID="01d62f7c-674b-4a61-a147-6ff35adc199a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj" Jul 17 21:53:30.952414 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:30.952379 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01d62f7c-674b-4a61-a147-6ff35adc199a-extensions-socket-volume\") pod \"01d62f7c-674b-4a61-a147-6ff35adc199a\" (UID: \"01d62f7c-674b-4a61-a147-6ff35adc199a\") " Jul 17 21:53:30.952621 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:30.952439 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhp29\" (UniqueName: \"kubernetes.io/projected/01d62f7c-674b-4a61-a147-6ff35adc199a-kube-api-access-fhp29\") pod \"01d62f7c-674b-4a61-a147-6ff35adc199a\" (UID: \"01d62f7c-674b-4a61-a147-6ff35adc199a\") " Jul 17 21:53:30.952719 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:30.952697 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d62f7c-674b-4a61-a147-6ff35adc199a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "01d62f7c-674b-4a61-a147-6ff35adc199a" (UID: "01d62f7c-674b-4a61-a147-6ff35adc199a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:53:30.954775 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:30.954749 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d62f7c-674b-4a61-a147-6ff35adc199a-kube-api-access-fhp29" (OuterVolumeSpecName: "kube-api-access-fhp29") pod "01d62f7c-674b-4a61-a147-6ff35adc199a" (UID: "01d62f7c-674b-4a61-a147-6ff35adc199a"). InnerVolumeSpecName "kube-api-access-fhp29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:53:31.053414 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:31.053381 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01d62f7c-674b-4a61-a147-6ff35adc199a-extensions-socket-volume\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:53:31.053414 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:31.053408 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhp29\" (UniqueName: \"kubernetes.io/projected/01d62f7c-674b-4a61-a147-6ff35adc199a-kube-api-access-fhp29\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:53:31.700695 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:31.700663 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01d62f7c-674b-4a61-a147-6ff35adc199a" path="/var/lib/kubelet/pods/01d62f7c-674b-4a61-a147-6ff35adc199a/volumes" Jul 17 21:53:31.876042 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:31.876014 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj" Jul 17 21:53:31.877961 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:53:31.877927 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cb2zj\" is forbidden: User \"system:node:ip-10-0-131-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-14.ec2.internal' and this object" podUID="01d62f7c-674b-4a61-a147-6ff35adc199a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj" Jul 17 21:53:31.879698 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:53:31.879673 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cb2zj\" is forbidden: User \"system:node:ip-10-0-131-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-14.ec2.internal' and this object" podUID="01d62f7c-674b-4a61-a147-6ff35adc199a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj" Jul 17 21:53:33.698570 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:53:33.698534 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-cb2zj\" is forbidden: User \"system:node:ip-10-0-131-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-14.ec2.internal' and this object" podUID="01d62f7c-674b-4a61-a147-6ff35adc199a" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-cb2zj" Jul 17 21:53:39.869553 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:39.869520 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr" Jul 17 21:53:39.869903 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:39.869589 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc" Jul 17 21:53:39.944438 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:39.944398 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr"] Jul 17 21:53:39.944626 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:39.944593 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr" podUID="7ccfa46a-90c8-4b49-998b-1dbc8eb498e9" containerName="manager" containerID="cri-o://d58b834ebb5ad06d5c1f78b1ad1debe7770fb6093ff19f421ea4e4ae064de839" gracePeriod=10 Jul 17 21:53:40.198924 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.198895 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr" Jul 17 21:53:40.208518 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.208472 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj"] Jul 17 21:53:40.208975 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.208953 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ccfa46a-90c8-4b49-998b-1dbc8eb498e9" containerName="manager" Jul 17 21:53:40.208975 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.208973 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ccfa46a-90c8-4b49-998b-1dbc8eb498e9" containerName="manager" Jul 17 21:53:40.209160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.209074 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ccfa46a-90c8-4b49-998b-1dbc8eb498e9" containerName="manager" Jul 17 21:53:40.211191 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.211173 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj" Jul 17 21:53:40.224129 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.224105 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj"] Jul 17 21:53:40.239044 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.239011 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t25hx\" (UniqueName: \"kubernetes.io/projected/7ccfa46a-90c8-4b49-998b-1dbc8eb498e9-kube-api-access-t25hx\") pod \"7ccfa46a-90c8-4b49-998b-1dbc8eb498e9\" (UID: \"7ccfa46a-90c8-4b49-998b-1dbc8eb498e9\") " Jul 17 21:53:40.239186 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.239067 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ccfa46a-90c8-4b49-998b-1dbc8eb498e9-extensions-socket-volume\") pod \"7ccfa46a-90c8-4b49-998b-1dbc8eb498e9\" (UID: \"7ccfa46a-90c8-4b49-998b-1dbc8eb498e9\") " Jul 17 21:53:40.239558 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.239521 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ccfa46a-90c8-4b49-998b-1dbc8eb498e9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7ccfa46a-90c8-4b49-998b-1dbc8eb498e9" (UID: "7ccfa46a-90c8-4b49-998b-1dbc8eb498e9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:53:40.241203 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.241170 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ccfa46a-90c8-4b49-998b-1dbc8eb498e9-kube-api-access-t25hx" (OuterVolumeSpecName: "kube-api-access-t25hx") pod "7ccfa46a-90c8-4b49-998b-1dbc8eb498e9" (UID: "7ccfa46a-90c8-4b49-998b-1dbc8eb498e9"). InnerVolumeSpecName "kube-api-access-t25hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:53:40.340081 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.340048 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx75j\" (UniqueName: \"kubernetes.io/projected/db1e2e3b-281f-4baa-bd34-c4d7c263e586-kube-api-access-cx75j\") pod \"kuadrant-operator-controller-manager-86878c769f-r72tj\" (UID: \"db1e2e3b-281f-4baa-bd34-c4d7c263e586\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj" Jul 17 21:53:40.340256 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.340106 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db1e2e3b-281f-4baa-bd34-c4d7c263e586-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-r72tj\" (UID: \"db1e2e3b-281f-4baa-bd34-c4d7c263e586\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj" Jul 17 21:53:40.340256 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.340153 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t25hx\" (UniqueName: \"kubernetes.io/projected/7ccfa46a-90c8-4b49-998b-1dbc8eb498e9-kube-api-access-t25hx\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:53:40.340256 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.340165 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7ccfa46a-90c8-4b49-998b-1dbc8eb498e9-extensions-socket-volume\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:53:40.440880 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.440840 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db1e2e3b-281f-4baa-bd34-c4d7c263e586-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-r72tj\" (UID: \"db1e2e3b-281f-4baa-bd34-c4d7c263e586\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj" Jul 17 21:53:40.441072 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.440914 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx75j\" (UniqueName: \"kubernetes.io/projected/db1e2e3b-281f-4baa-bd34-c4d7c263e586-kube-api-access-cx75j\") pod \"kuadrant-operator-controller-manager-86878c769f-r72tj\" (UID: \"db1e2e3b-281f-4baa-bd34-c4d7c263e586\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj" Jul 17 21:53:40.441226 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.441203 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db1e2e3b-281f-4baa-bd34-c4d7c263e586-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-r72tj\" (UID: \"db1e2e3b-281f-4baa-bd34-c4d7c263e586\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj" Jul 17 21:53:40.449776 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.449747 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx75j\" (UniqueName: \"kubernetes.io/projected/db1e2e3b-281f-4baa-bd34-c4d7c263e586-kube-api-access-cx75j\") pod \"kuadrant-operator-controller-manager-86878c769f-r72tj\" (UID: \"db1e2e3b-281f-4baa-bd34-c4d7c263e586\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj" Jul 17 21:53:40.522573 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.522540 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj" Jul 17 21:53:40.664081 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.664056 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj"] Jul 17 21:53:40.666238 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:53:40.666213 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1e2e3b_281f_4baa_bd34_c4d7c263e586.slice/crio-28059f7a7a035cf9b002925e893f78a18c4427aac1d50f334a66fe2c11eeefab WatchSource:0}: Error finding container 28059f7a7a035cf9b002925e893f78a18c4427aac1d50f334a66fe2c11eeefab: Status 404 returned error can't find the container with id 28059f7a7a035cf9b002925e893f78a18c4427aac1d50f334a66fe2c11eeefab Jul 17 21:53:40.920928 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.920888 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj" event={"ID":"db1e2e3b-281f-4baa-bd34-c4d7c263e586","Type":"ContainerStarted","Data":"a1381f02d20c3a136ed3e5e13c12a5cc9816f30a71fed9754d4218372c954f5d"} Jul 17 21:53:40.921317 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.920952 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj" event={"ID":"db1e2e3b-281f-4baa-bd34-c4d7c263e586","Type":"ContainerStarted","Data":"28059f7a7a035cf9b002925e893f78a18c4427aac1d50f334a66fe2c11eeefab"} Jul 17 21:53:40.921317 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.920984 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj" Jul 17 21:53:40.922216 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.922197 2577 generic.go:356] "Generic (PLEG): container finished" podID="7ccfa46a-90c8-4b49-998b-1dbc8eb498e9" containerID="d58b834ebb5ad06d5c1f78b1ad1debe7770fb6093ff19f421ea4e4ae064de839" exitCode=0 Jul 17 21:53:40.922281 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.922227 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr" event={"ID":"7ccfa46a-90c8-4b49-998b-1dbc8eb498e9","Type":"ContainerDied","Data":"d58b834ebb5ad06d5c1f78b1ad1debe7770fb6093ff19f421ea4e4ae064de839"} Jul 17 21:53:40.922281 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.922235 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr" Jul 17 21:53:40.922281 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.922255 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr" event={"ID":"7ccfa46a-90c8-4b49-998b-1dbc8eb498e9","Type":"ContainerDied","Data":"7f438c86b17d367c532d170a92c5750a74656d6e657922e09e40080e27d0f2a4"} Jul 17 21:53:40.922281 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.922277 2577 scope.go:117] "RemoveContainer" containerID="d58b834ebb5ad06d5c1f78b1ad1debe7770fb6093ff19f421ea4e4ae064de839" Jul 17 21:53:40.931394 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.931379 2577 scope.go:117] "RemoveContainer" containerID="d58b834ebb5ad06d5c1f78b1ad1debe7770fb6093ff19f421ea4e4ae064de839" Jul 17 21:53:40.931691 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:53:40.931672 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58b834ebb5ad06d5c1f78b1ad1debe7770fb6093ff19f421ea4e4ae064de839\": container with ID starting with d58b834ebb5ad06d5c1f78b1ad1debe7770fb6093ff19f421ea4e4ae064de839 not found: ID does not exist" containerID="d58b834ebb5ad06d5c1f78b1ad1debe7770fb6093ff19f421ea4e4ae064de839" Jul 17 21:53:40.931784 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.931696 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58b834ebb5ad06d5c1f78b1ad1debe7770fb6093ff19f421ea4e4ae064de839"} err="failed to get container status \"d58b834ebb5ad06d5c1f78b1ad1debe7770fb6093ff19f421ea4e4ae064de839\": rpc error: code = NotFound desc = could not find container \"d58b834ebb5ad06d5c1f78b1ad1debe7770fb6093ff19f421ea4e4ae064de839\": container with ID starting with d58b834ebb5ad06d5c1f78b1ad1debe7770fb6093ff19f421ea4e4ae064de839 not found: ID does not exist" Jul 17 21:53:40.962400 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.962321 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj" podStartSLOduration=0.962305881 podStartE2EDuration="962.305881ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:40.961082137 +0000 UTC m=+457.821305638" watchObservedRunningTime="2026-07-17 21:53:40.962305881 +0000 UTC m=+457.822529384" Jul 17 21:53:40.988478 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.988444 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr"] Jul 17 21:53:40.999942 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:40.999918 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qcfdr"] Jul 17 21:53:41.700827 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:41.700794 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ccfa46a-90c8-4b49-998b-1dbc8eb498e9" path="/var/lib/kubelet/pods/7ccfa46a-90c8-4b49-998b-1dbc8eb498e9/volumes" Jul 17 21:53:51.932025 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:51.931989 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj" Jul 17 21:53:51.980648 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:51.980622 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc"] Jul 17 21:53:51.980845 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:51.980825 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc" podUID="58d3303d-4dc0-4308-ba8c-b21d35c07408" containerName="manager" containerID="cri-o://99c8d2bd30846492590d0cab0a6784b5d32403a41ff3f57cdf31251d4a54739c" gracePeriod=10 Jul 17 21:53:52.826573 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:52.826547 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc" Jul 17 21:53:52.951833 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:52.951808 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhvdp\" (UniqueName: \"kubernetes.io/projected/58d3303d-4dc0-4308-ba8c-b21d35c07408-kube-api-access-rhvdp\") pod \"58d3303d-4dc0-4308-ba8c-b21d35c07408\" (UID: \"58d3303d-4dc0-4308-ba8c-b21d35c07408\") " Jul 17 21:53:52.952278 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:52.951879 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/58d3303d-4dc0-4308-ba8c-b21d35c07408-extensions-socket-volume\") pod \"58d3303d-4dc0-4308-ba8c-b21d35c07408\" (UID: \"58d3303d-4dc0-4308-ba8c-b21d35c07408\") " Jul 17 21:53:52.952340 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:52.952292 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58d3303d-4dc0-4308-ba8c-b21d35c07408-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "58d3303d-4dc0-4308-ba8c-b21d35c07408" (UID: "58d3303d-4dc0-4308-ba8c-b21d35c07408"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 21:53:52.953988 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:52.953960 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d3303d-4dc0-4308-ba8c-b21d35c07408-kube-api-access-rhvdp" (OuterVolumeSpecName: "kube-api-access-rhvdp") pod "58d3303d-4dc0-4308-ba8c-b21d35c07408" (UID: "58d3303d-4dc0-4308-ba8c-b21d35c07408"). InnerVolumeSpecName "kube-api-access-rhvdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:53:52.987218 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:52.987188 2577 generic.go:356] "Generic (PLEG): container finished" podID="58d3303d-4dc0-4308-ba8c-b21d35c07408" containerID="99c8d2bd30846492590d0cab0a6784b5d32403a41ff3f57cdf31251d4a54739c" exitCode=0 Jul 17 21:53:52.987327 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:52.987237 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc" Jul 17 21:53:52.987374 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:52.987237 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc" event={"ID":"58d3303d-4dc0-4308-ba8c-b21d35c07408","Type":"ContainerDied","Data":"99c8d2bd30846492590d0cab0a6784b5d32403a41ff3f57cdf31251d4a54739c"} Jul 17 21:53:52.987428 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:52.987372 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc" event={"ID":"58d3303d-4dc0-4308-ba8c-b21d35c07408","Type":"ContainerDied","Data":"4521f87bdf92c9522912e311f8df3c70c7efcd16452e0f71dbdf3bf3cec8792b"} Jul 17 21:53:52.987428 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:52.987401 2577 scope.go:117] "RemoveContainer" containerID="99c8d2bd30846492590d0cab0a6784b5d32403a41ff3f57cdf31251d4a54739c" Jul 17 21:53:52.997380 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:52.997364 2577 scope.go:117] "RemoveContainer" containerID="99c8d2bd30846492590d0cab0a6784b5d32403a41ff3f57cdf31251d4a54739c" Jul 17 21:53:52.997645 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:53:52.997625 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99c8d2bd30846492590d0cab0a6784b5d32403a41ff3f57cdf31251d4a54739c\": container with ID starting with 99c8d2bd30846492590d0cab0a6784b5d32403a41ff3f57cdf31251d4a54739c not found: ID does not exist" containerID="99c8d2bd30846492590d0cab0a6784b5d32403a41ff3f57cdf31251d4a54739c" Jul 17 21:53:52.997728 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:52.997654 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99c8d2bd30846492590d0cab0a6784b5d32403a41ff3f57cdf31251d4a54739c"} err="failed to get container status \"99c8d2bd30846492590d0cab0a6784b5d32403a41ff3f57cdf31251d4a54739c\": rpc error: code = NotFound desc = could not find container \"99c8d2bd30846492590d0cab0a6784b5d32403a41ff3f57cdf31251d4a54739c\": container with ID starting with 99c8d2bd30846492590d0cab0a6784b5d32403a41ff3f57cdf31251d4a54739c not found: ID does not exist" Jul 17 21:53:53.008959 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:53.008939 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc"] Jul 17 21:53:53.012986 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:53.012966 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rwmkc"] Jul 17 21:53:53.053205 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:53.053179 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhvdp\" (UniqueName: \"kubernetes.io/projected/58d3303d-4dc0-4308-ba8c-b21d35c07408-kube-api-access-rhvdp\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:53:53.053205 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:53.053205 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/58d3303d-4dc0-4308-ba8c-b21d35c07408-extensions-socket-volume\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:53:53.702511 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:53.702459 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58d3303d-4dc0-4308-ba8c-b21d35c07408" path="/var/lib/kubelet/pods/58d3303d-4dc0-4308-ba8c-b21d35c07408/volumes" Jul 17 21:53:54.332476 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:53:54.332442 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65fffd6797-j4mqh"] Jul 17 21:54:12.890457 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:12.890419 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f5qz8"] Jul 17 21:54:12.891741 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:12.891708 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58d3303d-4dc0-4308-ba8c-b21d35c07408" containerName="manager" Jul 17 21:54:12.891912 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:12.891899 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="58d3303d-4dc0-4308-ba8c-b21d35c07408" containerName="manager" Jul 17 21:54:12.892171 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:12.892160 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="58d3303d-4dc0-4308-ba8c-b21d35c07408" containerName="manager" Jul 17 21:54:12.954306 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:12.954277 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f5qz8"] Jul 17 21:54:12.954467 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:12.954388 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5qz8" Jul 17 21:54:12.956926 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:12.956906 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 17 21:54:12.985322 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:12.985295 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f5qz8"] Jul 17 21:54:13.038234 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:13.038211 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd86s\" (UniqueName: \"kubernetes.io/projected/7d76de34-6e4f-40d4-9335-463fc67e2668-kube-api-access-dd86s\") pod \"limitador-limitador-64c94fd64f-f5qz8\" (UID: \"7d76de34-6e4f-40d4-9335-463fc67e2668\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f5qz8" Jul 17 21:54:13.038373 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:13.038281 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d76de34-6e4f-40d4-9335-463fc67e2668-config-file\") pod \"limitador-limitador-64c94fd64f-f5qz8\" (UID: \"7d76de34-6e4f-40d4-9335-463fc67e2668\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f5qz8" Jul 17 21:54:13.139070 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:13.139040 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d76de34-6e4f-40d4-9335-463fc67e2668-config-file\") pod \"limitador-limitador-64c94fd64f-f5qz8\" (UID: \"7d76de34-6e4f-40d4-9335-463fc67e2668\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f5qz8" Jul 17 21:54:13.139239 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:13.139149 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd86s\" (UniqueName: \"kubernetes.io/projected/7d76de34-6e4f-40d4-9335-463fc67e2668-kube-api-access-dd86s\") pod \"limitador-limitador-64c94fd64f-f5qz8\" (UID: \"7d76de34-6e4f-40d4-9335-463fc67e2668\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f5qz8" Jul 17 21:54:13.139721 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:13.139703 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d76de34-6e4f-40d4-9335-463fc67e2668-config-file\") pod \"limitador-limitador-64c94fd64f-f5qz8\" (UID: \"7d76de34-6e4f-40d4-9335-463fc67e2668\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f5qz8" Jul 17 21:54:13.148028 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:13.147975 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd86s\" (UniqueName: \"kubernetes.io/projected/7d76de34-6e4f-40d4-9335-463fc67e2668-kube-api-access-dd86s\") pod \"limitador-limitador-64c94fd64f-f5qz8\" (UID: \"7d76de34-6e4f-40d4-9335-463fc67e2668\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-f5qz8" Jul 17 21:54:13.267809 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:13.267772 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5qz8" Jul 17 21:54:13.403577 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:13.403552 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f5qz8"] Jul 17 21:54:13.405033 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:54:13.405009 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d76de34_6e4f_40d4_9335_463fc67e2668.slice/crio-953f2e2a4110573a15bd32e09a75a303afaf8f7be75e9557f64c570db4422931 WatchSource:0}: Error finding container 953f2e2a4110573a15bd32e09a75a303afaf8f7be75e9557f64c570db4422931: Status 404 returned error can't find the container with id 953f2e2a4110573a15bd32e09a75a303afaf8f7be75e9557f64c570db4422931 Jul 17 21:54:14.100247 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:14.100157 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5qz8" event={"ID":"7d76de34-6e4f-40d4-9335-463fc67e2668","Type":"ContainerStarted","Data":"953f2e2a4110573a15bd32e09a75a303afaf8f7be75e9557f64c570db4422931"} Jul 17 21:54:17.116728 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:17.116693 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5qz8" event={"ID":"7d76de34-6e4f-40d4-9335-463fc67e2668","Type":"ContainerStarted","Data":"ccdcc03d8484ac0e42e5b2283b3f7bb8075c682d9d43c300cd5b9ea1398fa50e"} Jul 17 21:54:17.117178 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:17.116810 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5qz8" Jul 17 21:54:17.135571 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:17.135525 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5qz8" podStartSLOduration=2.448543301 podStartE2EDuration="5.135511446s" totalImagesPullingTime="2.686968145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:54:17.13351288 +0000 UTC m=+493.993736374" watchObservedRunningTime="2026-07-17 21:54:17.135511446 +0000 UTC m=+493.995734946" Jul 17 21:54:19.353964 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.353877 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65fffd6797-j4mqh" podUID="930695d4-4edb-4c35-bdeb-6b90356ba153" containerName="console" containerID="cri-o://f330d92bc530f1f23e7218044cdf7fc6be81329fe8b9f4c74ef5829a0881ae5d" gracePeriod=15 Jul 17 21:54:19.612587 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.612521 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:54:19.695477 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.695442 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/930695d4-4edb-4c35-bdeb-6b90356ba153-console-serving-cert\") pod \"930695d4-4edb-4c35-bdeb-6b90356ba153\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " Jul 17 21:54:19.695690 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.695534 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-console-config\") pod \"930695d4-4edb-4c35-bdeb-6b90356ba153\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " Jul 17 21:54:19.695690 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.695583 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-oauth-serving-cert\") pod \"930695d4-4edb-4c35-bdeb-6b90356ba153\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " Jul 17 21:54:19.695690 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.695611 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-trusted-ca-bundle\") pod \"930695d4-4edb-4c35-bdeb-6b90356ba153\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " Jul 17 21:54:19.695861 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.695709 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/930695d4-4edb-4c35-bdeb-6b90356ba153-console-oauth-config\") pod \"930695d4-4edb-4c35-bdeb-6b90356ba153\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " Jul 17 21:54:19.695861 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.695736 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbg2b\" (UniqueName: \"kubernetes.io/projected/930695d4-4edb-4c35-bdeb-6b90356ba153-kube-api-access-wbg2b\") pod \"930695d4-4edb-4c35-bdeb-6b90356ba153\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " Jul 17 21:54:19.695861 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.695794 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-service-ca\") pod \"930695d4-4edb-4c35-bdeb-6b90356ba153\" (UID: \"930695d4-4edb-4c35-bdeb-6b90356ba153\") " Jul 17 21:54:19.696035 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.696005 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "930695d4-4edb-4c35-bdeb-6b90356ba153" (UID: "930695d4-4edb-4c35-bdeb-6b90356ba153"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:54:19.696107 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.696017 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-console-config" (OuterVolumeSpecName: "console-config") pod "930695d4-4edb-4c35-bdeb-6b90356ba153" (UID: "930695d4-4edb-4c35-bdeb-6b90356ba153"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:54:19.696163 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.696097 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "930695d4-4edb-4c35-bdeb-6b90356ba153" (UID: "930695d4-4edb-4c35-bdeb-6b90356ba153"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:54:19.696387 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.696348 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-service-ca" (OuterVolumeSpecName: "service-ca") pod "930695d4-4edb-4c35-bdeb-6b90356ba153" (UID: "930695d4-4edb-4c35-bdeb-6b90356ba153"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:54:19.698176 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.698153 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/930695d4-4edb-4c35-bdeb-6b90356ba153-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "930695d4-4edb-4c35-bdeb-6b90356ba153" (UID: "930695d4-4edb-4c35-bdeb-6b90356ba153"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:54:19.698470 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.698444 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/930695d4-4edb-4c35-bdeb-6b90356ba153-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "930695d4-4edb-4c35-bdeb-6b90356ba153" (UID: "930695d4-4edb-4c35-bdeb-6b90356ba153"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:54:19.698470 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.698457 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930695d4-4edb-4c35-bdeb-6b90356ba153-kube-api-access-wbg2b" (OuterVolumeSpecName: "kube-api-access-wbg2b") pod "930695d4-4edb-4c35-bdeb-6b90356ba153" (UID: "930695d4-4edb-4c35-bdeb-6b90356ba153"). InnerVolumeSpecName "kube-api-access-wbg2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:54:19.796769 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.796737 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:54:19.796769 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.796760 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/930695d4-4edb-4c35-bdeb-6b90356ba153-console-serving-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:54:19.796769 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.796770 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-console-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:54:19.796974 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.796779 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-oauth-serving-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:54:19.796974 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.796786 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/930695d4-4edb-4c35-bdeb-6b90356ba153-trusted-ca-bundle\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:54:19.796974 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.796794 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/930695d4-4edb-4c35-bdeb-6b90356ba153-console-oauth-config\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:54:19.796974 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:19.796811 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbg2b\" (UniqueName: \"kubernetes.io/projected/930695d4-4edb-4c35-bdeb-6b90356ba153-kube-api-access-wbg2b\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:54:20.133220 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:20.133187 2577 generic.go:356] "Generic (PLEG): container finished" podID="930695d4-4edb-4c35-bdeb-6b90356ba153" containerID="f330d92bc530f1f23e7218044cdf7fc6be81329fe8b9f4c74ef5829a0881ae5d" exitCode=0 Jul 17 21:54:20.133220 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:20.133222 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65fffd6797-j4mqh" Jul 17 21:54:20.133448 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:20.133268 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fffd6797-j4mqh" event={"ID":"930695d4-4edb-4c35-bdeb-6b90356ba153","Type":"ContainerDied","Data":"f330d92bc530f1f23e7218044cdf7fc6be81329fe8b9f4c74ef5829a0881ae5d"} Jul 17 21:54:20.133448 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:20.133315 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65fffd6797-j4mqh" event={"ID":"930695d4-4edb-4c35-bdeb-6b90356ba153","Type":"ContainerDied","Data":"4ae8da4cf33d36d606b3e4ab28633249a7fa906bc686668087ac47ffafad4afc"} Jul 17 21:54:20.133448 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:20.133337 2577 scope.go:117] "RemoveContainer" containerID="f330d92bc530f1f23e7218044cdf7fc6be81329fe8b9f4c74ef5829a0881ae5d" Jul 17 21:54:20.142186 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:20.142169 2577 scope.go:117] "RemoveContainer" containerID="f330d92bc530f1f23e7218044cdf7fc6be81329fe8b9f4c74ef5829a0881ae5d" Jul 17 21:54:20.142456 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:54:20.142435 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f330d92bc530f1f23e7218044cdf7fc6be81329fe8b9f4c74ef5829a0881ae5d\": container with ID starting with f330d92bc530f1f23e7218044cdf7fc6be81329fe8b9f4c74ef5829a0881ae5d not found: ID does not exist" containerID="f330d92bc530f1f23e7218044cdf7fc6be81329fe8b9f4c74ef5829a0881ae5d" Jul 17 21:54:20.142565 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:20.142463 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f330d92bc530f1f23e7218044cdf7fc6be81329fe8b9f4c74ef5829a0881ae5d"} err="failed to get container status \"f330d92bc530f1f23e7218044cdf7fc6be81329fe8b9f4c74ef5829a0881ae5d\": rpc error: code = NotFound desc = could not find container \"f330d92bc530f1f23e7218044cdf7fc6be81329fe8b9f4c74ef5829a0881ae5d\": container with ID starting with f330d92bc530f1f23e7218044cdf7fc6be81329fe8b9f4c74ef5829a0881ae5d not found: ID does not exist" Jul 17 21:54:20.150402 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:20.150374 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65fffd6797-j4mqh"] Jul 17 21:54:20.156134 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:20.156113 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65fffd6797-j4mqh"] Jul 17 21:54:21.701152 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:21.701116 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="930695d4-4edb-4c35-bdeb-6b90356ba153" path="/var/lib/kubelet/pods/930695d4-4edb-4c35-bdeb-6b90356ba153/volumes" Jul 17 21:54:27.440006 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:27.439963 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f5qz8"] Jul 17 21:54:27.440436 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:27.440176 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5qz8" podUID="7d76de34-6e4f-40d4-9335-463fc67e2668" containerName="limitador" containerID="cri-o://ccdcc03d8484ac0e42e5b2283b3f7bb8075c682d9d43c300cd5b9ea1398fa50e" gracePeriod=30 Jul 17 21:54:27.440962 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:27.440942 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5qz8" Jul 17 21:54:27.996622 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:27.996558 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5qz8" Jul 17 21:54:28.067595 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:28.067570 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dd86s\" (UniqueName: \"kubernetes.io/projected/7d76de34-6e4f-40d4-9335-463fc67e2668-kube-api-access-dd86s\") pod \"7d76de34-6e4f-40d4-9335-463fc67e2668\" (UID: \"7d76de34-6e4f-40d4-9335-463fc67e2668\") " Jul 17 21:54:28.067735 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:28.067642 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d76de34-6e4f-40d4-9335-463fc67e2668-config-file\") pod \"7d76de34-6e4f-40d4-9335-463fc67e2668\" (UID: \"7d76de34-6e4f-40d4-9335-463fc67e2668\") " Jul 17 21:54:28.067946 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:28.067916 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d76de34-6e4f-40d4-9335-463fc67e2668-config-file" (OuterVolumeSpecName: "config-file") pod "7d76de34-6e4f-40d4-9335-463fc67e2668" (UID: "7d76de34-6e4f-40d4-9335-463fc67e2668"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 21:54:28.069702 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:28.069682 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d76de34-6e4f-40d4-9335-463fc67e2668-kube-api-access-dd86s" (OuterVolumeSpecName: "kube-api-access-dd86s") pod "7d76de34-6e4f-40d4-9335-463fc67e2668" (UID: "7d76de34-6e4f-40d4-9335-463fc67e2668"). InnerVolumeSpecName "kube-api-access-dd86s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:54:28.169094 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:28.169061 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd86s\" (UniqueName: \"kubernetes.io/projected/7d76de34-6e4f-40d4-9335-463fc67e2668-kube-api-access-dd86s\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:54:28.169094 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:28.169088 2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d76de34-6e4f-40d4-9335-463fc67e2668-config-file\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:54:28.174153 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:28.174126 2577 generic.go:356] "Generic (PLEG): container finished" podID="7d76de34-6e4f-40d4-9335-463fc67e2668" containerID="ccdcc03d8484ac0e42e5b2283b3f7bb8075c682d9d43c300cd5b9ea1398fa50e" exitCode=0 Jul 17 21:54:28.174278 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:28.174170 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5qz8" event={"ID":"7d76de34-6e4f-40d4-9335-463fc67e2668","Type":"ContainerDied","Data":"ccdcc03d8484ac0e42e5b2283b3f7bb8075c682d9d43c300cd5b9ea1398fa50e"} Jul 17 21:54:28.174278 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:28.174188 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5qz8" Jul 17 21:54:28.174278 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:28.174236 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-f5qz8" event={"ID":"7d76de34-6e4f-40d4-9335-463fc67e2668","Type":"ContainerDied","Data":"953f2e2a4110573a15bd32e09a75a303afaf8f7be75e9557f64c570db4422931"} Jul 17 21:54:28.174278 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:28.174258 2577 scope.go:117] "RemoveContainer" containerID="ccdcc03d8484ac0e42e5b2283b3f7bb8075c682d9d43c300cd5b9ea1398fa50e" Jul 17 21:54:28.183308 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:28.183292 2577 scope.go:117] "RemoveContainer" containerID="ccdcc03d8484ac0e42e5b2283b3f7bb8075c682d9d43c300cd5b9ea1398fa50e" Jul 17 21:54:28.183569 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:54:28.183549 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccdcc03d8484ac0e42e5b2283b3f7bb8075c682d9d43c300cd5b9ea1398fa50e\": container with ID starting with ccdcc03d8484ac0e42e5b2283b3f7bb8075c682d9d43c300cd5b9ea1398fa50e not found: ID does not exist" containerID="ccdcc03d8484ac0e42e5b2283b3f7bb8075c682d9d43c300cd5b9ea1398fa50e" Jul 17 21:54:28.183627 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:28.183577 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccdcc03d8484ac0e42e5b2283b3f7bb8075c682d9d43c300cd5b9ea1398fa50e"} err="failed to get container status \"ccdcc03d8484ac0e42e5b2283b3f7bb8075c682d9d43c300cd5b9ea1398fa50e\": rpc error: code = NotFound desc = could not find container \"ccdcc03d8484ac0e42e5b2283b3f7bb8075c682d9d43c300cd5b9ea1398fa50e\": container with ID starting with ccdcc03d8484ac0e42e5b2283b3f7bb8075c682d9d43c300cd5b9ea1398fa50e not found: ID does not exist" Jul 17 21:54:28.196103 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:28.196084 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f5qz8"] Jul 17 21:54:28.198690 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:28.198669 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-f5qz8"] Jul 17 21:54:29.701123 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:29.701087 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d76de34-6e4f-40d4-9335-463fc67e2668" path="/var/lib/kubelet/pods/7d76de34-6e4f-40d4-9335-463fc67e2668/volumes" Jul 17 21:54:30.220128 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.220096 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qhw5s"] Jul 17 21:54:30.220483 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.220461 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="930695d4-4edb-4c35-bdeb-6b90356ba153" containerName="console" Jul 17 21:54:30.220483 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.220484 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="930695d4-4edb-4c35-bdeb-6b90356ba153" containerName="console" Jul 17 21:54:30.220624 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.220521 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d76de34-6e4f-40d4-9335-463fc67e2668" containerName="limitador" Jul 17 21:54:30.220624 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.220530 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d76de34-6e4f-40d4-9335-463fc67e2668" containerName="limitador" Jul 17 21:54:30.220624 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.220621 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d76de34-6e4f-40d4-9335-463fc67e2668" containerName="limitador" Jul 17 21:54:30.220764 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.220638 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="930695d4-4edb-4c35-bdeb-6b90356ba153" containerName="console" Jul 17 21:54:30.223677 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.223658 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhw5s" Jul 17 21:54:30.226127 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.226106 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 17 21:54:30.226848 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.226831 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 17 21:54:30.226848 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.226842 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 17 21:54:30.226956 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.226842 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-sddmj\"" Jul 17 21:54:30.234782 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.234758 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qhw5s"] Jul 17 21:54:30.288707 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.288681 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5q2w\" (UniqueName: \"kubernetes.io/projected/c719223e-d7c1-440c-a434-19d91e6df9a6-kube-api-access-l5q2w\") pod \"postgres-69fb4594cf-qhw5s\" (UID: \"c719223e-d7c1-440c-a434-19d91e6df9a6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhw5s" Jul 17 21:54:30.288858 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.288813 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c719223e-d7c1-440c-a434-19d91e6df9a6-data\") pod \"postgres-69fb4594cf-qhw5s\" (UID: \"c719223e-d7c1-440c-a434-19d91e6df9a6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhw5s" Jul 17 21:54:30.390026 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.389996 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c719223e-d7c1-440c-a434-19d91e6df9a6-data\") pod \"postgres-69fb4594cf-qhw5s\" (UID: \"c719223e-d7c1-440c-a434-19d91e6df9a6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhw5s" Jul 17 21:54:30.390197 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.390042 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5q2w\" (UniqueName: \"kubernetes.io/projected/c719223e-d7c1-440c-a434-19d91e6df9a6-kube-api-access-l5q2w\") pod \"postgres-69fb4594cf-qhw5s\" (UID: \"c719223e-d7c1-440c-a434-19d91e6df9a6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhw5s" Jul 17 21:54:30.390404 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.390382 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c719223e-d7c1-440c-a434-19d91e6df9a6-data\") pod \"postgres-69fb4594cf-qhw5s\" (UID: \"c719223e-d7c1-440c-a434-19d91e6df9a6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhw5s" Jul 17 21:54:30.398692 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.398670 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5q2w\" (UniqueName: \"kubernetes.io/projected/c719223e-d7c1-440c-a434-19d91e6df9a6-kube-api-access-l5q2w\") pod \"postgres-69fb4594cf-qhw5s\" (UID: \"c719223e-d7c1-440c-a434-19d91e6df9a6\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhw5s" Jul 17 21:54:30.534252 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.534160 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhw5s" Jul 17 21:54:30.670753 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:30.670726 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qhw5s"] Jul 17 21:54:30.672963 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:54:30.672933 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc719223e_d7c1_440c_a434_19d91e6df9a6.slice/crio-7c62b62e3123404815f1561d84ae597a4ea15b5f433dda701d30a5272c3b85ac WatchSource:0}: Error finding container 7c62b62e3123404815f1561d84ae597a4ea15b5f433dda701d30a5272c3b85ac: Status 404 returned error can't find the container with id 7c62b62e3123404815f1561d84ae597a4ea15b5f433dda701d30a5272c3b85ac Jul 17 21:54:31.190770 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:31.190737 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhw5s" event={"ID":"c719223e-d7c1-440c-a434-19d91e6df9a6","Type":"ContainerStarted","Data":"7c62b62e3123404815f1561d84ae597a4ea15b5f433dda701d30a5272c3b85ac"} Jul 17 21:54:36.222354 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:36.222316 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhw5s" event={"ID":"c719223e-d7c1-440c-a434-19d91e6df9a6","Type":"ContainerStarted","Data":"38ca69404db2c3b2d277f3ccc9880ffcc5647840cd4644a9e61eb49aa3598540"} Jul 17 21:54:36.222772 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:36.222460 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhw5s" Jul 17 21:54:36.239994 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:36.239957 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhw5s" podStartSLOduration=1.4154721719999999 podStartE2EDuration="6.239945047s" totalImagesPullingTime="4.824472875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:54:36.23885887 +0000 UTC m=+513.099082392" watchObservedRunningTime="2026-07-17 21:54:36.239945047 +0000 UTC m=+513.100168549" Jul 17 21:54:42.256830 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:42.256790 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qhw5s" Jul 17 21:54:49.210921 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.210881 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9f458cd86-6jkxw"] Jul 17 21:54:49.234413 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.234385 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9f458cd86-6jkxw"] Jul 17 21:54:49.234613 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.234526 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9f458cd86-6jkxw" Jul 17 21:54:49.237287 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.237252 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 17 21:54:49.237287 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.237281 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 17 21:54:49.237511 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.237361 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-s695q\"" Jul 17 21:54:49.355739 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.355702 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b-webhook-cert\") pod \"maas-controller-9f458cd86-6jkxw\" (UID: \"7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b\") " pod="opendatahub/maas-controller-9f458cd86-6jkxw" Jul 17 21:54:49.355921 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.355778 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm8gz\" (UniqueName: \"kubernetes.io/projected/7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b-kube-api-access-vm8gz\") pod \"maas-controller-9f458cd86-6jkxw\" (UID: \"7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b\") " pod="opendatahub/maas-controller-9f458cd86-6jkxw" Jul 17 21:54:49.381816 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.381788 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6fbf6969b9-ktbl9"] Jul 17 21:54:49.384392 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.384374 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fbf6969b9-ktbl9" Jul 17 21:54:49.395792 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.395768 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fbf6969b9-ktbl9"] Jul 17 21:54:49.456242 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.456206 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b-webhook-cert\") pod \"maas-controller-9f458cd86-6jkxw\" (UID: \"7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b\") " pod="opendatahub/maas-controller-9f458cd86-6jkxw" Jul 17 21:54:49.456417 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.456271 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm8gz\" (UniqueName: \"kubernetes.io/projected/7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b-kube-api-access-vm8gz\") pod \"maas-controller-9f458cd86-6jkxw\" (UID: \"7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b\") " pod="opendatahub/maas-controller-9f458cd86-6jkxw" Jul 17 21:54:49.456417 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:54:49.456351 2577 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 17 21:54:49.456630 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:54:49.456423 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b-webhook-cert podName:7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b nodeName:}" failed. No retries permitted until 2026-07-17 21:54:49.956406025 +0000 UTC m=+526.816629511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b-webhook-cert") pod "maas-controller-9f458cd86-6jkxw" (UID: "7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b") : secret "maas-controller-webhook-cert" not found Jul 17 21:54:49.465298 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.465239 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm8gz\" (UniqueName: \"kubernetes.io/projected/7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b-kube-api-access-vm8gz\") pod \"maas-controller-9f458cd86-6jkxw\" (UID: \"7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b\") " pod="opendatahub/maas-controller-9f458cd86-6jkxw" Jul 17 21:54:49.557824 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.557783 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkc4x\" (UniqueName: \"kubernetes.io/projected/665df72e-fe50-4a5c-9c7f-6b24e053afa9-kube-api-access-bkc4x\") pod \"maas-controller-6fbf6969b9-ktbl9\" (UID: \"665df72e-fe50-4a5c-9c7f-6b24e053afa9\") " pod="opendatahub/maas-controller-6fbf6969b9-ktbl9" Jul 17 21:54:49.558017 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.557853 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/665df72e-fe50-4a5c-9c7f-6b24e053afa9-webhook-cert\") pod \"maas-controller-6fbf6969b9-ktbl9\" (UID: \"665df72e-fe50-4a5c-9c7f-6b24e053afa9\") " pod="opendatahub/maas-controller-6fbf6969b9-ktbl9" Jul 17 21:54:49.659159 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.659127 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkc4x\" (UniqueName: \"kubernetes.io/projected/665df72e-fe50-4a5c-9c7f-6b24e053afa9-kube-api-access-bkc4x\") pod \"maas-controller-6fbf6969b9-ktbl9\" (UID: \"665df72e-fe50-4a5c-9c7f-6b24e053afa9\") " pod="opendatahub/maas-controller-6fbf6969b9-ktbl9" Jul 17 21:54:49.659327 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.659189 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/665df72e-fe50-4a5c-9c7f-6b24e053afa9-webhook-cert\") pod \"maas-controller-6fbf6969b9-ktbl9\" (UID: \"665df72e-fe50-4a5c-9c7f-6b24e053afa9\") " pod="opendatahub/maas-controller-6fbf6969b9-ktbl9" Jul 17 21:54:49.661761 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.661736 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/665df72e-fe50-4a5c-9c7f-6b24e053afa9-webhook-cert\") pod \"maas-controller-6fbf6969b9-ktbl9\" (UID: \"665df72e-fe50-4a5c-9c7f-6b24e053afa9\") " pod="opendatahub/maas-controller-6fbf6969b9-ktbl9" Jul 17 21:54:49.666762 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.666739 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkc4x\" (UniqueName: \"kubernetes.io/projected/665df72e-fe50-4a5c-9c7f-6b24e053afa9-kube-api-access-bkc4x\") pod \"maas-controller-6fbf6969b9-ktbl9\" (UID: \"665df72e-fe50-4a5c-9c7f-6b24e053afa9\") " pod="opendatahub/maas-controller-6fbf6969b9-ktbl9" Jul 17 21:54:49.695094 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.695075 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fbf6969b9-ktbl9" Jul 17 21:54:49.962192 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.962147 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b-webhook-cert\") pod \"maas-controller-9f458cd86-6jkxw\" (UID: \"7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b\") " pod="opendatahub/maas-controller-9f458cd86-6jkxw" Jul 17 21:54:49.964747 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:49.964727 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b-webhook-cert\") pod \"maas-controller-9f458cd86-6jkxw\" (UID: \"7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b\") " pod="opendatahub/maas-controller-9f458cd86-6jkxw" Jul 17 21:54:50.037668 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:50.035064 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fbf6969b9-ktbl9"] Jul 17 21:54:50.145130 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:50.145097 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9f458cd86-6jkxw" Jul 17 21:54:50.275245 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:50.275220 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9f458cd86-6jkxw"] Jul 17 21:54:50.276268 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:54:50.276248 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b51d4e5_d6ce_4ac9_b3df_ffbfced79e3b.slice/crio-fbc1af562c4f2c91a87226da23910b8a572870facab2e1ff99d795590628d96b WatchSource:0}: Error finding container fbc1af562c4f2c91a87226da23910b8a572870facab2e1ff99d795590628d96b: Status 404 returned error can't find the container with id fbc1af562c4f2c91a87226da23910b8a572870facab2e1ff99d795590628d96b Jul 17 21:54:50.291109 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:50.291035 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9f458cd86-6jkxw" event={"ID":"7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b","Type":"ContainerStarted","Data":"fbc1af562c4f2c91a87226da23910b8a572870facab2e1ff99d795590628d96b"} Jul 17 21:54:50.292388 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:50.292366 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fbf6969b9-ktbl9" event={"ID":"665df72e-fe50-4a5c-9c7f-6b24e053afa9","Type":"ContainerStarted","Data":"565536ee78a903c19672bb585e84528345e365f8d2492017fbc97a5b7a1a5311"} Jul 17 21:54:53.313533 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:53.313474 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9f458cd86-6jkxw" event={"ID":"7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b","Type":"ContainerStarted","Data":"f82efc07d0886dbc3928ba5fb379f8ad112a631dc8db727f616275c354123353"} Jul 17 21:54:53.314006 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:53.313601 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9f458cd86-6jkxw" Jul 17 21:54:53.315055 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:53.315036 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fbf6969b9-ktbl9" event={"ID":"665df72e-fe50-4a5c-9c7f-6b24e053afa9","Type":"ContainerStarted","Data":"a56d34fefbe1f1620d2e717d7edbc50857675c1df5dbf52a8c7ea137cac17a27"} Jul 17 21:54:53.315130 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:53.315084 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6fbf6969b9-ktbl9" Jul 17 21:54:53.330732 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:53.330686 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-9f458cd86-6jkxw" podStartSLOduration=1.830651059 podStartE2EDuration="4.330672081s" totalImagesPullingTime="2.500021022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:54:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:54:53.329598079 +0000 UTC m=+530.189821605" watchObservedRunningTime="2026-07-17 21:54:53.330672081 +0000 UTC m=+530.190895583" Jul 17 21:54:53.350452 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:54:53.350386 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6fbf6969b9-ktbl9" podStartSLOduration=1.6209562709999998 podStartE2EDuration="4.350369902s" totalImagesPullingTime="2.729413631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:54:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:54:53.347376269 +0000 UTC m=+530.207599839" watchObservedRunningTime="2026-07-17 21:54:53.350369902 +0000 UTC m=+530.210593405" Jul 17 21:55:04.322057 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:04.322025 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9f458cd86-6jkxw" Jul 17 21:55:04.322446 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:04.322092 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6fbf6969b9-ktbl9" Jul 17 21:55:04.382931 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:04.382897 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9f458cd86-6jkxw"] Jul 17 21:55:04.383131 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:04.383078 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-9f458cd86-6jkxw" podUID="7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b" containerName="manager" containerID="cri-o://f82efc07d0886dbc3928ba5fb379f8ad112a631dc8db727f616275c354123353" gracePeriod=30 Jul 17 21:55:04.645425 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:04.645396 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9f458cd86-6jkxw" Jul 17 21:55:04.691299 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:04.691264 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vm8gz\" (UniqueName: \"kubernetes.io/projected/7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b-kube-api-access-vm8gz\") pod \"7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b\" (UID: \"7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b\") " Jul 17 21:55:04.691464 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:04.691324 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b-webhook-cert\") pod \"7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b\" (UID: \"7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b\") " Jul 17 21:55:04.694209 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:04.694180 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b" (UID: "7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:55:04.694325 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:04.694249 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b-kube-api-access-vm8gz" (OuterVolumeSpecName: "kube-api-access-vm8gz") pod "7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b" (UID: "7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b"). InnerVolumeSpecName "kube-api-access-vm8gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:55:04.792005 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:04.791964 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm8gz\" (UniqueName: \"kubernetes.io/projected/7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b-kube-api-access-vm8gz\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:55:04.792005 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:04.791994 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b-webhook-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:55:05.376665 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:05.376628 2577 generic.go:356] "Generic (PLEG): container finished" podID="7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b" containerID="f82efc07d0886dbc3928ba5fb379f8ad112a631dc8db727f616275c354123353" exitCode=0 Jul 17 21:55:05.377136 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:05.376697 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9f458cd86-6jkxw" Jul 17 21:55:05.377136 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:05.376705 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9f458cd86-6jkxw" event={"ID":"7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b","Type":"ContainerDied","Data":"f82efc07d0886dbc3928ba5fb379f8ad112a631dc8db727f616275c354123353"} Jul 17 21:55:05.377136 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:05.376763 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9f458cd86-6jkxw" event={"ID":"7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b","Type":"ContainerDied","Data":"fbc1af562c4f2c91a87226da23910b8a572870facab2e1ff99d795590628d96b"} Jul 17 21:55:05.377136 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:05.376797 2577 scope.go:117] "RemoveContainer" containerID="f82efc07d0886dbc3928ba5fb379f8ad112a631dc8db727f616275c354123353" Jul 17 21:55:05.386475 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:05.386453 2577 scope.go:117] "RemoveContainer" containerID="f82efc07d0886dbc3928ba5fb379f8ad112a631dc8db727f616275c354123353" Jul 17 21:55:05.386793 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:55:05.386774 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f82efc07d0886dbc3928ba5fb379f8ad112a631dc8db727f616275c354123353\": container with ID starting with f82efc07d0886dbc3928ba5fb379f8ad112a631dc8db727f616275c354123353 not found: ID does not exist" containerID="f82efc07d0886dbc3928ba5fb379f8ad112a631dc8db727f616275c354123353" Jul 17 21:55:05.386847 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:05.386803 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f82efc07d0886dbc3928ba5fb379f8ad112a631dc8db727f616275c354123353"} err="failed to get container status \"f82efc07d0886dbc3928ba5fb379f8ad112a631dc8db727f616275c354123353\": rpc error: code = NotFound desc = could not find container \"f82efc07d0886dbc3928ba5fb379f8ad112a631dc8db727f616275c354123353\": container with ID starting with f82efc07d0886dbc3928ba5fb379f8ad112a631dc8db727f616275c354123353 not found: ID does not exist" Jul 17 21:55:05.399033 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:05.399000 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9f458cd86-6jkxw"] Jul 17 21:55:05.402802 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:05.402778 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-9f458cd86-6jkxw"] Jul 17 21:55:05.700561 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:05.700532 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b" path="/var/lib/kubelet/pods/7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b/volumes" Jul 17 21:55:25.425619 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.425588 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7786b868b8-2zdbb"] Jul 17 21:55:25.426068 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.426037 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b" containerName="manager" Jul 17 21:55:25.426068 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.426052 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b" containerName="manager" Jul 17 21:55:25.426173 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.426125 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b51d4e5-d6ce-4ac9-b3df-ffbfced79e3b" containerName="manager" Jul 17 21:55:25.428853 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.428835 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7786b868b8-2zdbb" Jul 17 21:55:25.431010 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.430987 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 17 21:55:25.431151 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.431136 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-vxq68\"" Jul 17 21:55:25.441060 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.441040 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7786b868b8-2zdbb"] Jul 17 21:55:25.574715 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.574679 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41a4c34-1541-4244-a4d7-8f90dae8ed52-maas-api-tls\") pod \"maas-api-7786b868b8-2zdbb\" (UID: \"b41a4c34-1541-4244-a4d7-8f90dae8ed52\") " pod="odh-ai-gateway-infra/maas-api-7786b868b8-2zdbb" Jul 17 21:55:25.574715 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.574715 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41a4c34-1541-4244-a4d7-8f90dae8ed52-openshift-service-ca\") pod \"maas-api-7786b868b8-2zdbb\" (UID: \"b41a4c34-1541-4244-a4d7-8f90dae8ed52\") " pod="odh-ai-gateway-infra/maas-api-7786b868b8-2zdbb" Jul 17 21:55:25.574918 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.574732 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxplj\" (UniqueName: \"kubernetes.io/projected/b41a4c34-1541-4244-a4d7-8f90dae8ed52-kube-api-access-dxplj\") pod \"maas-api-7786b868b8-2zdbb\" (UID: \"b41a4c34-1541-4244-a4d7-8f90dae8ed52\") " pod="odh-ai-gateway-infra/maas-api-7786b868b8-2zdbb" Jul 17 21:55:25.675564 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.675530 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41a4c34-1541-4244-a4d7-8f90dae8ed52-maas-api-tls\") pod \"maas-api-7786b868b8-2zdbb\" (UID: \"b41a4c34-1541-4244-a4d7-8f90dae8ed52\") " pod="odh-ai-gateway-infra/maas-api-7786b868b8-2zdbb" Jul 17 21:55:25.675782 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.675572 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41a4c34-1541-4244-a4d7-8f90dae8ed52-openshift-service-ca\") pod \"maas-api-7786b868b8-2zdbb\" (UID: \"b41a4c34-1541-4244-a4d7-8f90dae8ed52\") " pod="odh-ai-gateway-infra/maas-api-7786b868b8-2zdbb" Jul 17 21:55:25.675782 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.675595 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxplj\" (UniqueName: \"kubernetes.io/projected/b41a4c34-1541-4244-a4d7-8f90dae8ed52-kube-api-access-dxplj\") pod \"maas-api-7786b868b8-2zdbb\" (UID: \"b41a4c34-1541-4244-a4d7-8f90dae8ed52\") " pod="odh-ai-gateway-infra/maas-api-7786b868b8-2zdbb" Jul 17 21:55:25.676310 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.676287 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41a4c34-1541-4244-a4d7-8f90dae8ed52-openshift-service-ca\") pod \"maas-api-7786b868b8-2zdbb\" (UID: \"b41a4c34-1541-4244-a4d7-8f90dae8ed52\") " pod="odh-ai-gateway-infra/maas-api-7786b868b8-2zdbb" Jul 17 21:55:25.678670 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.678647 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41a4c34-1541-4244-a4d7-8f90dae8ed52-maas-api-tls\") pod \"maas-api-7786b868b8-2zdbb\" (UID: \"b41a4c34-1541-4244-a4d7-8f90dae8ed52\") " pod="odh-ai-gateway-infra/maas-api-7786b868b8-2zdbb" Jul 17 21:55:25.683884 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.683861 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxplj\" (UniqueName: \"kubernetes.io/projected/b41a4c34-1541-4244-a4d7-8f90dae8ed52-kube-api-access-dxplj\") pod \"maas-api-7786b868b8-2zdbb\" (UID: \"b41a4c34-1541-4244-a4d7-8f90dae8ed52\") " pod="odh-ai-gateway-infra/maas-api-7786b868b8-2zdbb" Jul 17 21:55:25.739916 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.739882 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7786b868b8-2zdbb" Jul 17 21:55:25.895774 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.895752 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7786b868b8-2zdbb"] Jul 17 21:55:25.898169 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:55:25.898137 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41a4c34_1541_4244_a4d7_8f90dae8ed52.slice/crio-53739f566bd994d0bd0828fc72b4866f0a8e4152efe4b5ad20f8b6724b7a0ce4 WatchSource:0}: Error finding container 53739f566bd994d0bd0828fc72b4866f0a8e4152efe4b5ad20f8b6724b7a0ce4: Status 404 returned error can't find the container with id 53739f566bd994d0bd0828fc72b4866f0a8e4152efe4b5ad20f8b6724b7a0ce4 Jul 17 21:55:25.960350 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.960322 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-nflxn"] Jul 17 21:55:25.998638 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.998596 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-nflxn"] Jul 17 21:55:25.998794 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.998666 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-lr4vw"] Jul 17 21:55:25.998794 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:25.998716 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-nflxn" Jul 17 21:55:26.001180 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.001156 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 17 21:55:26.001561 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.001542 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-qsm4d\"" Jul 17 21:55:26.002317 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.002292 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 17 21:55:26.002479 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.002458 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 17 21:55:26.028434 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.028414 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-lr4vw"] Jul 17 21:55:26.028571 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.028522 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-lr4vw" Jul 17 21:55:26.079522 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.079464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5zbm\" (UniqueName: \"kubernetes.io/projected/aadcfd02-e238-4f21-a88e-5481899e72bb-kube-api-access-n5zbm\") pod \"payload-pre-processing-7677b774bb-nflxn\" (UID: \"aadcfd02-e238-4f21-a88e-5481899e72bb\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-nflxn" Jul 17 21:55:26.079683 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.079545 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aadcfd02-e238-4f21-a88e-5481899e72bb-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-nflxn\" (UID: \"aadcfd02-e238-4f21-a88e-5481899e72bb\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-nflxn" Jul 17 21:55:26.180300 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.180268 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5zbm\" (UniqueName: \"kubernetes.io/projected/aadcfd02-e238-4f21-a88e-5481899e72bb-kube-api-access-n5zbm\") pod \"payload-pre-processing-7677b774bb-nflxn\" (UID: \"aadcfd02-e238-4f21-a88e-5481899e72bb\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-nflxn" Jul 17 21:55:26.180467 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.180326 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aadcfd02-e238-4f21-a88e-5481899e72bb-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-nflxn\" (UID: \"aadcfd02-e238-4f21-a88e-5481899e72bb\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-nflxn" Jul 17 21:55:26.180467 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.180395 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9157197c-0cda-4d3d-b031-7863f59defe1-plugins-config-volume\") pod \"payload-processing-5bf47cd988-lr4vw\" (UID: \"9157197c-0cda-4d3d-b031-7863f59defe1\") " pod="openshift-ingress/payload-processing-5bf47cd988-lr4vw" Jul 17 21:55:26.180467 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.180449 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qtds\" (UniqueName: \"kubernetes.io/projected/9157197c-0cda-4d3d-b031-7863f59defe1-kube-api-access-2qtds\") pod \"payload-processing-5bf47cd988-lr4vw\" (UID: \"9157197c-0cda-4d3d-b031-7863f59defe1\") " pod="openshift-ingress/payload-processing-5bf47cd988-lr4vw" Jul 17 21:55:26.181112 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.181086 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aadcfd02-e238-4f21-a88e-5481899e72bb-plugins-config-volume\") pod \"payload-pre-processing-7677b774bb-nflxn\" (UID: \"aadcfd02-e238-4f21-a88e-5481899e72bb\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-nflxn" Jul 17 21:55:26.188343 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.188320 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5zbm\" (UniqueName: \"kubernetes.io/projected/aadcfd02-e238-4f21-a88e-5481899e72bb-kube-api-access-n5zbm\") pod \"payload-pre-processing-7677b774bb-nflxn\" (UID: \"aadcfd02-e238-4f21-a88e-5481899e72bb\") " pod="openshift-ingress/payload-pre-processing-7677b774bb-nflxn" Jul 17 21:55:26.281830 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.281755 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9157197c-0cda-4d3d-b031-7863f59defe1-plugins-config-volume\") pod \"payload-processing-5bf47cd988-lr4vw\" (UID: \"9157197c-0cda-4d3d-b031-7863f59defe1\") " pod="openshift-ingress/payload-processing-5bf47cd988-lr4vw" Jul 17 21:55:26.281830 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.281805 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qtds\" (UniqueName: \"kubernetes.io/projected/9157197c-0cda-4d3d-b031-7863f59defe1-kube-api-access-2qtds\") pod \"payload-processing-5bf47cd988-lr4vw\" (UID: \"9157197c-0cda-4d3d-b031-7863f59defe1\") " pod="openshift-ingress/payload-processing-5bf47cd988-lr4vw" Jul 17 21:55:26.282419 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.282399 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9157197c-0cda-4d3d-b031-7863f59defe1-plugins-config-volume\") pod \"payload-processing-5bf47cd988-lr4vw\" (UID: \"9157197c-0cda-4d3d-b031-7863f59defe1\") " pod="openshift-ingress/payload-processing-5bf47cd988-lr4vw" Jul 17 21:55:26.289684 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.289657 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qtds\" (UniqueName: \"kubernetes.io/projected/9157197c-0cda-4d3d-b031-7863f59defe1-kube-api-access-2qtds\") pod \"payload-processing-5bf47cd988-lr4vw\" (UID: \"9157197c-0cda-4d3d-b031-7863f59defe1\") " pod="openshift-ingress/payload-processing-5bf47cd988-lr4vw" Jul 17 21:55:26.311235 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.311211 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7677b774bb-nflxn" Jul 17 21:55:26.337816 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.337777 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5bf47cd988-lr4vw" Jul 17 21:55:26.478660 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.478637 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7677b774bb-nflxn"] Jul 17 21:55:26.480417 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:55:26.480380 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadcfd02_e238_4f21_a88e_5481899e72bb.slice/crio-5a725f075ecf6a9237297472d69bae3a5e9fb4bfefa232571cb06d36efa65468 WatchSource:0}: Error finding container 5a725f075ecf6a9237297472d69bae3a5e9fb4bfefa232571cb06d36efa65468: Status 404 returned error can't find the container with id 5a725f075ecf6a9237297472d69bae3a5e9fb4bfefa232571cb06d36efa65468 Jul 17 21:55:26.482411 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.482383 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7786b868b8-2zdbb" event={"ID":"b41a4c34-1541-4244-a4d7-8f90dae8ed52","Type":"ContainerStarted","Data":"53739f566bd994d0bd0828fc72b4866f0a8e4152efe4b5ad20f8b6724b7a0ce4"} Jul 17 21:55:26.497577 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:26.497555 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5bf47cd988-lr4vw"] Jul 17 21:55:26.499029 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:55:26.499007 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9157197c_0cda_4d3d_b031_7863f59defe1.slice/crio-8ae6d210eccc3f7805df908c7bf415075ecf819fe320d485d81c6af1c12628a2 WatchSource:0}: Error finding container 8ae6d210eccc3f7805df908c7bf415075ecf819fe320d485d81c6af1c12628a2: Status 404 returned error can't find the container with id 8ae6d210eccc3f7805df908c7bf415075ecf819fe320d485d81c6af1c12628a2 Jul 17 21:55:27.495579 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:27.495537 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-lr4vw" event={"ID":"9157197c-0cda-4d3d-b031-7863f59defe1","Type":"ContainerStarted","Data":"8ae6d210eccc3f7805df908c7bf415075ecf819fe320d485d81c6af1c12628a2"} Jul 17 21:55:27.498862 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:27.498782 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-nflxn" event={"ID":"aadcfd02-e238-4f21-a88e-5481899e72bb","Type":"ContainerStarted","Data":"5a725f075ecf6a9237297472d69bae3a5e9fb4bfefa232571cb06d36efa65468"} Jul 17 21:55:28.505424 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:28.505399 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7786b868b8-2zdbb" event={"ID":"b41a4c34-1541-4244-a4d7-8f90dae8ed52","Type":"ContainerStarted","Data":"77e27c7ee17dee249bddf7b827e80eb8cb35c4457beddf96b2b24da4316de0e3"} Jul 17 21:55:28.505813 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:28.505510 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7786b868b8-2zdbb" Jul 17 21:55:28.527549 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:28.527509 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7786b868b8-2zdbb" podStartSLOduration=1.440032464 podStartE2EDuration="3.527469023s" totalImagesPullingTime="2.087436559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:55:28.526618946 +0000 UTC m=+565.386842471" watchObservedRunningTime="2026-07-17 21:55:28.527469023 +0000 UTC m=+565.387692525" Jul 17 21:55:29.511589 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:29.511549 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7677b774bb-nflxn" event={"ID":"aadcfd02-e238-4f21-a88e-5481899e72bb","Type":"ContainerStarted","Data":"806548a6a76ef256eb85ed13b2ab93bfe54aef0c0dc3010077085d4db2eec542"} Jul 17 21:55:29.512015 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:29.511635 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-nflxn" Jul 17 21:55:29.513308 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:29.513286 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5bf47cd988-lr4vw" event={"ID":"9157197c-0cda-4d3d-b031-7863f59defe1","Type":"ContainerStarted","Data":"dc99e3c601863e0aaf0a25c4892f1cf349e2ced7bb56a77763d308c38da80e7a"} Jul 17 21:55:29.513430 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:29.513410 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5bf47cd988-lr4vw" Jul 17 21:55:29.527638 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:29.527598 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7677b774bb-nflxn" podStartSLOduration=2.554058945 podStartE2EDuration="4.527581096s" totalImagesPullingTime="1.973522151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:55:29.525868803 +0000 UTC m=+566.386092306" watchObservedRunningTime="2026-07-17 21:55:29.527581096 +0000 UTC m=+566.387804668" Jul 17 21:55:29.540578 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:29.540535 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5bf47cd988-lr4vw" podStartSLOduration=2.578874635 podStartE2EDuration="4.54052185s" totalImagesPullingTime="1.961647215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:55:29.539404274 +0000 UTC m=+566.399627799" watchObservedRunningTime="2026-07-17 21:55:29.54052185 +0000 UTC m=+566.400745352" Jul 17 21:55:34.518891 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:34.518864 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7786b868b8-2zdbb" Jul 17 21:55:40.521303 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.521278 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7677b774bb-nflxn" Jul 17 21:55:40.521734 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.521347 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5bf47cd988-lr4vw" Jul 17 21:55:40.525057 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.525031 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5"] Jul 17 21:55:40.528842 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.528821 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.531301 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.531280 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 17 21:55:40.531746 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.531725 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7nlsc\"" Jul 17 21:55:40.531848 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.531753 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 17 21:55:40.532000 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.531986 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 17 21:55:40.538651 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.538612 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5"] Jul 17 21:55:40.712991 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.712958 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bd87f8-9f90-4c08-83a0-a199e45a9114-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5\" (UID: \"d6bd87f8-9f90-4c08-83a0-a199e45a9114\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.713160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.712998 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6bd87f8-9f90-4c08-83a0-a199e45a9114-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5\" (UID: \"d6bd87f8-9f90-4c08-83a0-a199e45a9114\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.713160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.713041 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6bd87f8-9f90-4c08-83a0-a199e45a9114-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5\" (UID: \"d6bd87f8-9f90-4c08-83a0-a199e45a9114\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.713160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.713082 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bd87f8-9f90-4c08-83a0-a199e45a9114-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5\" (UID: \"d6bd87f8-9f90-4c08-83a0-a199e45a9114\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.713160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.713116 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c54gz\" (UniqueName: \"kubernetes.io/projected/d6bd87f8-9f90-4c08-83a0-a199e45a9114-kube-api-access-c54gz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5\" (UID: \"d6bd87f8-9f90-4c08-83a0-a199e45a9114\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.713160 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.713146 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6bd87f8-9f90-4c08-83a0-a199e45a9114-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5\" (UID: \"d6bd87f8-9f90-4c08-83a0-a199e45a9114\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.814381 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.814295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6bd87f8-9f90-4c08-83a0-a199e45a9114-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5\" (UID: \"d6bd87f8-9f90-4c08-83a0-a199e45a9114\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.814381 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.814340 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bd87f8-9f90-4c08-83a0-a199e45a9114-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5\" (UID: \"d6bd87f8-9f90-4c08-83a0-a199e45a9114\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.814606 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.814466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c54gz\" (UniqueName: \"kubernetes.io/projected/d6bd87f8-9f90-4c08-83a0-a199e45a9114-kube-api-access-c54gz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5\" (UID: \"d6bd87f8-9f90-4c08-83a0-a199e45a9114\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.814606 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.814564 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6bd87f8-9f90-4c08-83a0-a199e45a9114-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5\" (UID: \"d6bd87f8-9f90-4c08-83a0-a199e45a9114\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.814693 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.814667 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bd87f8-9f90-4c08-83a0-a199e45a9114-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5\" (UID: \"d6bd87f8-9f90-4c08-83a0-a199e45a9114\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.814743 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.814716 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6bd87f8-9f90-4c08-83a0-a199e45a9114-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5\" (UID: \"d6bd87f8-9f90-4c08-83a0-a199e45a9114\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.814840 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.814808 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6bd87f8-9f90-4c08-83a0-a199e45a9114-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5\" (UID: \"d6bd87f8-9f90-4c08-83a0-a199e45a9114\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.815009 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.814989 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6bd87f8-9f90-4c08-83a0-a199e45a9114-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5\" (UID: \"d6bd87f8-9f90-4c08-83a0-a199e45a9114\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.815091 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.815010 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6bd87f8-9f90-4c08-83a0-a199e45a9114-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5\" (UID: \"d6bd87f8-9f90-4c08-83a0-a199e45a9114\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.816741 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.816724 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6bd87f8-9f90-4c08-83a0-a199e45a9114-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5\" (UID: \"d6bd87f8-9f90-4c08-83a0-a199e45a9114\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.817054 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.817028 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6bd87f8-9f90-4c08-83a0-a199e45a9114-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5\" (UID: \"d6bd87f8-9f90-4c08-83a0-a199e45a9114\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.822229 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.822209 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c54gz\" (UniqueName: \"kubernetes.io/projected/d6bd87f8-9f90-4c08-83a0-a199e45a9114-kube-api-access-c54gz\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5\" (UID: \"d6bd87f8-9f90-4c08-83a0-a199e45a9114\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.840189 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.840168 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:40.983051 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:40.982951 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5"] Jul 17 21:55:40.985938 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:55:40.985909 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6bd87f8_9f90_4c08_83a0_a199e45a9114.slice/crio-e4a1cd17fc2e2e8f41dc50d6ad20832cde9864b4e1865f8efbce03b91b668f7f WatchSource:0}: Error finding container e4a1cd17fc2e2e8f41dc50d6ad20832cde9864b4e1865f8efbce03b91b668f7f: Status 404 returned error can't find the container with id e4a1cd17fc2e2e8f41dc50d6ad20832cde9864b4e1865f8efbce03b91b668f7f Jul 17 21:55:41.579137 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:41.579108 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" event={"ID":"d6bd87f8-9f90-4c08-83a0-a199e45a9114","Type":"ContainerStarted","Data":"e4a1cd17fc2e2e8f41dc50d6ad20832cde9864b4e1865f8efbce03b91b668f7f"} Jul 17 21:55:42.585840 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:42.585801 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" event={"ID":"d6bd87f8-9f90-4c08-83a0-a199e45a9114","Type":"ContainerStarted","Data":"948125ae1648abc27d1fb7bc4510c56378a51991b3d476707626054599609f0c"} Jul 17 21:55:42.602135 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:42.602086 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" podStartSLOduration=1.433584844 podStartE2EDuration="2.602071802s" totalImagesPullingTime="1.168486958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:55:42.601108068 +0000 UTC m=+579.461331595" watchObservedRunningTime="2026-07-17 21:55:42.602071802 +0000 UTC m=+579.462295303" Jul 17 21:55:50.840607 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:50.840573 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:50.841021 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:50.840882 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:50.853601 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:50.853579 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:51.641990 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:51.641965 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5" Jul 17 21:55:59.224590 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.224545 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm"] Jul 17 21:55:59.250077 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.250045 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm"] Jul 17 21:55:59.250406 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.250386 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.253127 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.253100 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 17 21:55:59.277346 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.277316 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b431df7c-d0d8-48cf-b94e-dbfb026e91af-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v49hm\" (UID: \"b431df7c-d0d8-48cf-b94e-dbfb026e91af\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.277521 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.277376 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbddw\" (UniqueName: \"kubernetes.io/projected/b431df7c-d0d8-48cf-b94e-dbfb026e91af-kube-api-access-wbddw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v49hm\" (UID: \"b431df7c-d0d8-48cf-b94e-dbfb026e91af\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.277521 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.277458 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b431df7c-d0d8-48cf-b94e-dbfb026e91af-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v49hm\" (UID: \"b431df7c-d0d8-48cf-b94e-dbfb026e91af\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.277521 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.277505 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b431df7c-d0d8-48cf-b94e-dbfb026e91af-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v49hm\" (UID: \"b431df7c-d0d8-48cf-b94e-dbfb026e91af\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.277702 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.277529 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b431df7c-d0d8-48cf-b94e-dbfb026e91af-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v49hm\" (UID: \"b431df7c-d0d8-48cf-b94e-dbfb026e91af\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.277702 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.277558 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b431df7c-d0d8-48cf-b94e-dbfb026e91af-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v49hm\" (UID: \"b431df7c-d0d8-48cf-b94e-dbfb026e91af\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.378942 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.378904 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b431df7c-d0d8-48cf-b94e-dbfb026e91af-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v49hm\" (UID: \"b431df7c-d0d8-48cf-b94e-dbfb026e91af\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.378942 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.378948 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b431df7c-d0d8-48cf-b94e-dbfb026e91af-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v49hm\" (UID: \"b431df7c-d0d8-48cf-b94e-dbfb026e91af\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.379162 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.378982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b431df7c-d0d8-48cf-b94e-dbfb026e91af-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v49hm\" (UID: \"b431df7c-d0d8-48cf-b94e-dbfb026e91af\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.379162 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.379023 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbddw\" (UniqueName: \"kubernetes.io/projected/b431df7c-d0d8-48cf-b94e-dbfb026e91af-kube-api-access-wbddw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v49hm\" (UID: \"b431df7c-d0d8-48cf-b94e-dbfb026e91af\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.379162 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.379059 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b431df7c-d0d8-48cf-b94e-dbfb026e91af-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v49hm\" (UID: \"b431df7c-d0d8-48cf-b94e-dbfb026e91af\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.379162 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.379080 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b431df7c-d0d8-48cf-b94e-dbfb026e91af-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v49hm\" (UID: \"b431df7c-d0d8-48cf-b94e-dbfb026e91af\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.379414 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.379389 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b431df7c-d0d8-48cf-b94e-dbfb026e91af-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v49hm\" (UID: \"b431df7c-d0d8-48cf-b94e-dbfb026e91af\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.379485 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.379427 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b431df7c-d0d8-48cf-b94e-dbfb026e91af-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v49hm\" (UID: \"b431df7c-d0d8-48cf-b94e-dbfb026e91af\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.379590 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.379531 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b431df7c-d0d8-48cf-b94e-dbfb026e91af-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v49hm\" (UID: \"b431df7c-d0d8-48cf-b94e-dbfb026e91af\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.381536 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.381513 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b431df7c-d0d8-48cf-b94e-dbfb026e91af-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v49hm\" (UID: \"b431df7c-d0d8-48cf-b94e-dbfb026e91af\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.381721 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.381708 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b431df7c-d0d8-48cf-b94e-dbfb026e91af-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v49hm\" (UID: \"b431df7c-d0d8-48cf-b94e-dbfb026e91af\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.387129 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.387108 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbddw\" (UniqueName: \"kubernetes.io/projected/b431df7c-d0d8-48cf-b94e-dbfb026e91af-kube-api-access-wbddw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-v49hm\" (UID: \"b431df7c-d0d8-48cf-b94e-dbfb026e91af\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.562774 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.562695 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:55:59.707381 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:55:59.707350 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb431df7c_d0d8_48cf_b94e_dbfb026e91af.slice/crio-e2e9f85184110f9435c9526a28b8cbc3ab8ab77b1458261f262a6b68faad4199 WatchSource:0}: Error finding container e2e9f85184110f9435c9526a28b8cbc3ab8ab77b1458261f262a6b68faad4199: Status 404 returned error can't find the container with id e2e9f85184110f9435c9526a28b8cbc3ab8ab77b1458261f262a6b68faad4199 Jul 17 21:55:59.710035 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:55:59.710010 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm"] Jul 17 21:56:00.677407 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:56:00.677371 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" event={"ID":"b431df7c-d0d8-48cf-b94e-dbfb026e91af","Type":"ContainerStarted","Data":"4f33fcf00eb9dacdb282c0b866b23a1a9159be7e03a5143eb63efc4f98bf1a03"} Jul 17 21:56:00.677807 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:56:00.677433 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" event={"ID":"b431df7c-d0d8-48cf-b94e-dbfb026e91af","Type":"ContainerStarted","Data":"e2e9f85184110f9435c9526a28b8cbc3ab8ab77b1458261f262a6b68faad4199"} Jul 17 21:56:00.696784 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:56:00.696732 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" podStartSLOduration=1.5109246710000002 podStartE2EDuration="1.696716622s" totalImagesPullingTime="185.791951ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:56:00.695136095 +0000 UTC m=+597.555359624" watchObservedRunningTime="2026-07-17 21:56:00.696716622 +0000 UTC m=+597.556940124" Jul 17 21:56:09.562978 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:56:09.562944 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:56:09.565402 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:56:09.563001 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:56:09.575182 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:56:09.575161 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:56:09.739506 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:56:09.739473 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-v49hm" Jul 17 21:58:46.587339 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:58:46.587306 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fbf6969b9-ktbl9"] Jul 17 21:58:46.587873 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:58:46.587535 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6fbf6969b9-ktbl9" podUID="665df72e-fe50-4a5c-9c7f-6b24e053afa9" containerName="manager" containerID="cri-o://a56d34fefbe1f1620d2e717d7edbc50857675c1df5dbf52a8c7ea137cac17a27" gracePeriod=30 Jul 17 21:58:46.844785 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:58:46.844727 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fbf6969b9-ktbl9" Jul 17 21:58:46.930187 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:58:46.930154 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkc4x\" (UniqueName: \"kubernetes.io/projected/665df72e-fe50-4a5c-9c7f-6b24e053afa9-kube-api-access-bkc4x\") pod \"665df72e-fe50-4a5c-9c7f-6b24e053afa9\" (UID: \"665df72e-fe50-4a5c-9c7f-6b24e053afa9\") " Jul 17 21:58:46.930386 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:58:46.930232 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/665df72e-fe50-4a5c-9c7f-6b24e053afa9-webhook-cert\") pod \"665df72e-fe50-4a5c-9c7f-6b24e053afa9\" (UID: \"665df72e-fe50-4a5c-9c7f-6b24e053afa9\") " Jul 17 21:58:46.932422 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:58:46.932394 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/665df72e-fe50-4a5c-9c7f-6b24e053afa9-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "665df72e-fe50-4a5c-9c7f-6b24e053afa9" (UID: "665df72e-fe50-4a5c-9c7f-6b24e053afa9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 21:58:46.932422 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:58:46.932398 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665df72e-fe50-4a5c-9c7f-6b24e053afa9-kube-api-access-bkc4x" (OuterVolumeSpecName: "kube-api-access-bkc4x") pod "665df72e-fe50-4a5c-9c7f-6b24e053afa9" (UID: "665df72e-fe50-4a5c-9c7f-6b24e053afa9"). InnerVolumeSpecName "kube-api-access-bkc4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 21:58:47.031255 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:58:47.031222 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkc4x\" (UniqueName: \"kubernetes.io/projected/665df72e-fe50-4a5c-9c7f-6b24e053afa9-kube-api-access-bkc4x\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:58:47.031255 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:58:47.031252 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/665df72e-fe50-4a5c-9c7f-6b24e053afa9-webhook-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 21:58:47.552147 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:58:47.552112 2577 generic.go:356] "Generic (PLEG): container finished" podID="665df72e-fe50-4a5c-9c7f-6b24e053afa9" containerID="a56d34fefbe1f1620d2e717d7edbc50857675c1df5dbf52a8c7ea137cac17a27" exitCode=0 Jul 17 21:58:47.552338 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:58:47.552168 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fbf6969b9-ktbl9" Jul 17 21:58:47.552338 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:58:47.552204 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fbf6969b9-ktbl9" event={"ID":"665df72e-fe50-4a5c-9c7f-6b24e053afa9","Type":"ContainerDied","Data":"a56d34fefbe1f1620d2e717d7edbc50857675c1df5dbf52a8c7ea137cac17a27"} Jul 17 21:58:47.552338 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:58:47.552260 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fbf6969b9-ktbl9" event={"ID":"665df72e-fe50-4a5c-9c7f-6b24e053afa9","Type":"ContainerDied","Data":"565536ee78a903c19672bb585e84528345e365f8d2492017fbc97a5b7a1a5311"} Jul 17 21:58:47.552338 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:58:47.552286 2577 scope.go:117] "RemoveContainer" containerID="a56d34fefbe1f1620d2e717d7edbc50857675c1df5dbf52a8c7ea137cac17a27" Jul 17 21:58:47.561740 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:58:47.561723 2577 scope.go:117] "RemoveContainer" containerID="a56d34fefbe1f1620d2e717d7edbc50857675c1df5dbf52a8c7ea137cac17a27" Jul 17 21:58:47.562022 ip-10-0-131-14 kubenswrapper[2577]: E0717 21:58:47.561999 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a56d34fefbe1f1620d2e717d7edbc50857675c1df5dbf52a8c7ea137cac17a27\": container with ID starting with a56d34fefbe1f1620d2e717d7edbc50857675c1df5dbf52a8c7ea137cac17a27 not found: ID does not exist" containerID="a56d34fefbe1f1620d2e717d7edbc50857675c1df5dbf52a8c7ea137cac17a27" Jul 17 21:58:47.562100 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:58:47.562036 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a56d34fefbe1f1620d2e717d7edbc50857675c1df5dbf52a8c7ea137cac17a27"} err="failed to get container status \"a56d34fefbe1f1620d2e717d7edbc50857675c1df5dbf52a8c7ea137cac17a27\": rpc error: code = NotFound desc = could not find container \"a56d34fefbe1f1620d2e717d7edbc50857675c1df5dbf52a8c7ea137cac17a27\": container with ID starting with a56d34fefbe1f1620d2e717d7edbc50857675c1df5dbf52a8c7ea137cac17a27 not found: ID does not exist" Jul 17 21:58:47.582364 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:58:47.582337 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fbf6969b9-ktbl9"] Jul 17 21:58:47.591609 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:58:47.591583 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6fbf6969b9-ktbl9"] Jul 17 21:58:47.701951 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:58:47.701916 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="665df72e-fe50-4a5c-9c7f-6b24e053afa9" path="/var/lib/kubelet/pods/665df72e-fe50-4a5c-9c7f-6b24e053afa9/volumes" Jul 17 21:59:48.500580 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:48.500481 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6fbf6969b9-8n688"] Jul 17 21:59:48.501058 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:48.500942 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="665df72e-fe50-4a5c-9c7f-6b24e053afa9" containerName="manager" Jul 17 21:59:48.501058 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:48.500954 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="665df72e-fe50-4a5c-9c7f-6b24e053afa9" containerName="manager" Jul 17 21:59:48.501058 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:48.501034 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="665df72e-fe50-4a5c-9c7f-6b24e053afa9" containerName="manager" Jul 17 21:59:48.504366 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:48.504350 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fbf6969b9-8n688" Jul 17 21:59:48.507383 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:48.507365 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 17 21:59:48.508289 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:48.508272 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 17 21:59:48.508289 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:48.508282 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-s695q\"" Jul 17 21:59:48.518210 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:48.518191 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fbf6969b9-8n688"] Jul 17 21:59:48.608266 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:48.608224 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghvn4\" (UniqueName: \"kubernetes.io/projected/2d9ff870-4613-441a-92e9-7256dd1d0ed4-kube-api-access-ghvn4\") pod \"maas-controller-6fbf6969b9-8n688\" (UID: \"2d9ff870-4613-441a-92e9-7256dd1d0ed4\") " pod="opendatahub/maas-controller-6fbf6969b9-8n688" Jul 17 21:59:48.608433 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:48.608287 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d9ff870-4613-441a-92e9-7256dd1d0ed4-webhook-cert\") pod \"maas-controller-6fbf6969b9-8n688\" (UID: \"2d9ff870-4613-441a-92e9-7256dd1d0ed4\") " pod="opendatahub/maas-controller-6fbf6969b9-8n688" Jul 17 21:59:48.709710 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:48.709679 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghvn4\" (UniqueName: \"kubernetes.io/projected/2d9ff870-4613-441a-92e9-7256dd1d0ed4-kube-api-access-ghvn4\") pod \"maas-controller-6fbf6969b9-8n688\" (UID: \"2d9ff870-4613-441a-92e9-7256dd1d0ed4\") " pod="opendatahub/maas-controller-6fbf6969b9-8n688" Jul 17 21:59:48.709871 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:48.709727 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d9ff870-4613-441a-92e9-7256dd1d0ed4-webhook-cert\") pod \"maas-controller-6fbf6969b9-8n688\" (UID: \"2d9ff870-4613-441a-92e9-7256dd1d0ed4\") " pod="opendatahub/maas-controller-6fbf6969b9-8n688" Jul 17 21:59:48.712191 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:48.712162 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d9ff870-4613-441a-92e9-7256dd1d0ed4-webhook-cert\") pod \"maas-controller-6fbf6969b9-8n688\" (UID: \"2d9ff870-4613-441a-92e9-7256dd1d0ed4\") " pod="opendatahub/maas-controller-6fbf6969b9-8n688" Jul 17 21:59:48.718424 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:48.718402 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghvn4\" (UniqueName: \"kubernetes.io/projected/2d9ff870-4613-441a-92e9-7256dd1d0ed4-kube-api-access-ghvn4\") pod \"maas-controller-6fbf6969b9-8n688\" (UID: \"2d9ff870-4613-441a-92e9-7256dd1d0ed4\") " pod="opendatahub/maas-controller-6fbf6969b9-8n688" Jul 17 21:59:48.814657 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:48.814584 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fbf6969b9-8n688" Jul 17 21:59:48.951791 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:48.951763 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fbf6969b9-8n688"] Jul 17 21:59:48.953756 ip-10-0-131-14 kubenswrapper[2577]: W0717 21:59:48.953724 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d9ff870_4613_441a_92e9_7256dd1d0ed4.slice/crio-f517766d88d1fed5aa17c6d15c3dfaa1d941c9c981be01ec0d5cefe792d31883 WatchSource:0}: Error finding container f517766d88d1fed5aa17c6d15c3dfaa1d941c9c981be01ec0d5cefe792d31883: Status 404 returned error can't find the container with id f517766d88d1fed5aa17c6d15c3dfaa1d941c9c981be01ec0d5cefe792d31883 Jul 17 21:59:48.955118 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:48.955100 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 21:59:49.875560 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:49.875520 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fbf6969b9-8n688" event={"ID":"2d9ff870-4613-441a-92e9-7256dd1d0ed4","Type":"ContainerStarted","Data":"6d780d65967aedb52e4d8d95777196f270b354f9f900442129bc45bfd120e792"} Jul 17 21:59:49.875936 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:49.875579 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fbf6969b9-8n688" event={"ID":"2d9ff870-4613-441a-92e9-7256dd1d0ed4","Type":"ContainerStarted","Data":"f517766d88d1fed5aa17c6d15c3dfaa1d941c9c981be01ec0d5cefe792d31883"} Jul 17 21:59:49.875936 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:49.875689 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6fbf6969b9-8n688" Jul 17 21:59:49.893889 ip-10-0-131-14 kubenswrapper[2577]: I0717 21:59:49.893845 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6fbf6969b9-8n688" podStartSLOduration=1.479424415 podStartE2EDuration="1.893828828s" totalImagesPullingTime="414.404413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:59:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:59:49.891716164 +0000 UTC m=+826.751939695" watchObservedRunningTime="2026-07-17 21:59:49.893828828 +0000 UTC m=+826.754052330" Jul 17 22:00:00.884463 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:00.884430 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6fbf6969b9-8n688" Jul 17 22:00:05.533399 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:05.533354 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fbf6969b9-8n688"] Jul 17 22:00:05.533918 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:05.533709 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6fbf6969b9-8n688" podUID="2d9ff870-4613-441a-92e9-7256dd1d0ed4" containerName="manager" containerID="cri-o://6d780d65967aedb52e4d8d95777196f270b354f9f900442129bc45bfd120e792" gracePeriod=30 Jul 17 22:00:05.786365 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:05.786301 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fbf6969b9-8n688" Jul 17 22:00:05.857936 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:05.857910 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghvn4\" (UniqueName: \"kubernetes.io/projected/2d9ff870-4613-441a-92e9-7256dd1d0ed4-kube-api-access-ghvn4\") pod \"2d9ff870-4613-441a-92e9-7256dd1d0ed4\" (UID: \"2d9ff870-4613-441a-92e9-7256dd1d0ed4\") " Jul 17 22:00:05.858120 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:05.857945 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d9ff870-4613-441a-92e9-7256dd1d0ed4-webhook-cert\") pod \"2d9ff870-4613-441a-92e9-7256dd1d0ed4\" (UID: \"2d9ff870-4613-441a-92e9-7256dd1d0ed4\") " Jul 17 22:00:05.860148 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:05.860120 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d9ff870-4613-441a-92e9-7256dd1d0ed4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2d9ff870-4613-441a-92e9-7256dd1d0ed4" (UID: "2d9ff870-4613-441a-92e9-7256dd1d0ed4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:00:05.860255 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:05.860159 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d9ff870-4613-441a-92e9-7256dd1d0ed4-kube-api-access-ghvn4" (OuterVolumeSpecName: "kube-api-access-ghvn4") pod "2d9ff870-4613-441a-92e9-7256dd1d0ed4" (UID: "2d9ff870-4613-441a-92e9-7256dd1d0ed4"). InnerVolumeSpecName "kube-api-access-ghvn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:00:05.958740 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:05.958707 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghvn4\" (UniqueName: \"kubernetes.io/projected/2d9ff870-4613-441a-92e9-7256dd1d0ed4-kube-api-access-ghvn4\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:00:05.958740 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:05.958739 2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d9ff870-4613-441a-92e9-7256dd1d0ed4-webhook-cert\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:00:05.962100 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:05.962074 2577 generic.go:356] "Generic (PLEG): container finished" podID="2d9ff870-4613-441a-92e9-7256dd1d0ed4" containerID="6d780d65967aedb52e4d8d95777196f270b354f9f900442129bc45bfd120e792" exitCode=0 Jul 17 22:00:05.962209 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:05.962119 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fbf6969b9-8n688" Jul 17 22:00:05.962209 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:05.962130 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fbf6969b9-8n688" event={"ID":"2d9ff870-4613-441a-92e9-7256dd1d0ed4","Type":"ContainerDied","Data":"6d780d65967aedb52e4d8d95777196f270b354f9f900442129bc45bfd120e792"} Jul 17 22:00:05.962209 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:05.962175 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fbf6969b9-8n688" event={"ID":"2d9ff870-4613-441a-92e9-7256dd1d0ed4","Type":"ContainerDied","Data":"f517766d88d1fed5aa17c6d15c3dfaa1d941c9c981be01ec0d5cefe792d31883"} Jul 17 22:00:05.962209 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:05.962195 2577 scope.go:117] "RemoveContainer" containerID="6d780d65967aedb52e4d8d95777196f270b354f9f900442129bc45bfd120e792" Jul 17 22:00:05.972093 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:05.972074 2577 scope.go:117] "RemoveContainer" containerID="6d780d65967aedb52e4d8d95777196f270b354f9f900442129bc45bfd120e792" Jul 17 22:00:05.972367 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:00:05.972349 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d780d65967aedb52e4d8d95777196f270b354f9f900442129bc45bfd120e792\": container with ID starting with 6d780d65967aedb52e4d8d95777196f270b354f9f900442129bc45bfd120e792 not found: ID does not exist" containerID="6d780d65967aedb52e4d8d95777196f270b354f9f900442129bc45bfd120e792" Jul 17 22:00:05.972431 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:05.972376 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d780d65967aedb52e4d8d95777196f270b354f9f900442129bc45bfd120e792"} err="failed to get container status \"6d780d65967aedb52e4d8d95777196f270b354f9f900442129bc45bfd120e792\": rpc error: code = NotFound desc = could not find container \"6d780d65967aedb52e4d8d95777196f270b354f9f900442129bc45bfd120e792\": container with ID starting with 6d780d65967aedb52e4d8d95777196f270b354f9f900442129bc45bfd120e792 not found: ID does not exist" Jul 17 22:00:05.983707 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:05.983685 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6fbf6969b9-8n688"] Jul 17 22:00:05.988710 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:05.988686 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6fbf6969b9-8n688"] Jul 17 22:00:06.775079 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:06.775042 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6fbf6969b9-dt6sj"] Jul 17 22:00:06.775649 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:06.775631 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d9ff870-4613-441a-92e9-7256dd1d0ed4" containerName="manager" Jul 17 22:00:06.775744 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:06.775651 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d9ff870-4613-441a-92e9-7256dd1d0ed4" containerName="manager" Jul 17 22:00:06.775805 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:06.775785 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d9ff870-4613-441a-92e9-7256dd1d0ed4" containerName="manager" Jul 17 22:00:06.780136 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:06.780116 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fbf6969b9-dt6sj" Jul 17 22:00:06.783183 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:06.783160 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 17 22:00:06.783183 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:06.783174 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 17 22:00:06.783543 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:06.783160 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-s695q\"" Jul 17 22:00:06.786693 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:06.786633 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fbf6969b9-dt6sj"] Jul 17 22:00:06.866864 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:06.866829 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld6k5\" (UniqueName: \"kubernetes.io/projected/1fce1ef7-da7c-4996-a7cd-4780f21286a5-kube-api-access-ld6k5\") pod \"maas-controller-6fbf6969b9-dt6sj\" (UID: \"1fce1ef7-da7c-4996-a7cd-4780f21286a5\") " pod="opendatahub/maas-controller-6fbf6969b9-dt6sj" Jul 17 22:00:06.867049 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:06.866975 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1fce1ef7-da7c-4996-a7cd-4780f21286a5-webhook-cert\") pod \"maas-controller-6fbf6969b9-dt6sj\" (UID: \"1fce1ef7-da7c-4996-a7cd-4780f21286a5\") " pod="opendatahub/maas-controller-6fbf6969b9-dt6sj" Jul 17 22:00:06.967976 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:06.967937 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1fce1ef7-da7c-4996-a7cd-4780f21286a5-webhook-cert\") pod \"maas-controller-6fbf6969b9-dt6sj\" (UID: \"1fce1ef7-da7c-4996-a7cd-4780f21286a5\") " pod="opendatahub/maas-controller-6fbf6969b9-dt6sj" Jul 17 22:00:06.968189 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:06.967999 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld6k5\" (UniqueName: \"kubernetes.io/projected/1fce1ef7-da7c-4996-a7cd-4780f21286a5-kube-api-access-ld6k5\") pod \"maas-controller-6fbf6969b9-dt6sj\" (UID: \"1fce1ef7-da7c-4996-a7cd-4780f21286a5\") " pod="opendatahub/maas-controller-6fbf6969b9-dt6sj" Jul 17 22:00:06.970674 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:06.970651 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1fce1ef7-da7c-4996-a7cd-4780f21286a5-webhook-cert\") pod \"maas-controller-6fbf6969b9-dt6sj\" (UID: \"1fce1ef7-da7c-4996-a7cd-4780f21286a5\") " pod="opendatahub/maas-controller-6fbf6969b9-dt6sj" Jul 17 22:00:06.975506 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:06.975472 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld6k5\" (UniqueName: \"kubernetes.io/projected/1fce1ef7-da7c-4996-a7cd-4780f21286a5-kube-api-access-ld6k5\") pod \"maas-controller-6fbf6969b9-dt6sj\" (UID: \"1fce1ef7-da7c-4996-a7cd-4780f21286a5\") " pod="opendatahub/maas-controller-6fbf6969b9-dt6sj" Jul 17 22:00:07.096550 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:07.096429 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6fbf6969b9-dt6sj" Jul 17 22:00:07.237226 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:07.237198 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6fbf6969b9-dt6sj"] Jul 17 22:00:07.239100 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:00:07.239075 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fce1ef7_da7c_4996_a7cd_4780f21286a5.slice/crio-6567625908a64cfa66e7adf8269f851b70dedc9aac0cff21893528e027b3d366 WatchSource:0}: Error finding container 6567625908a64cfa66e7adf8269f851b70dedc9aac0cff21893528e027b3d366: Status 404 returned error can't find the container with id 6567625908a64cfa66e7adf8269f851b70dedc9aac0cff21893528e027b3d366 Jul 17 22:00:07.701400 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:07.701365 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d9ff870-4613-441a-92e9-7256dd1d0ed4" path="/var/lib/kubelet/pods/2d9ff870-4613-441a-92e9-7256dd1d0ed4/volumes" Jul 17 22:00:07.975586 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:07.975551 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fbf6969b9-dt6sj" event={"ID":"1fce1ef7-da7c-4996-a7cd-4780f21286a5","Type":"ContainerStarted","Data":"2c2f543428a2cf508134de701adf4474acf38a889c2486320ff1aad86e61abf0"} Jul 17 22:00:07.975972 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:07.975615 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6fbf6969b9-dt6sj" event={"ID":"1fce1ef7-da7c-4996-a7cd-4780f21286a5","Type":"ContainerStarted","Data":"6567625908a64cfa66e7adf8269f851b70dedc9aac0cff21893528e027b3d366"} Jul 17 22:00:07.975972 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:07.975699 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6fbf6969b9-dt6sj" Jul 17 22:00:07.992454 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:07.992407 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6fbf6969b9-dt6sj" podStartSLOduration=1.640143798 podStartE2EDuration="1.992392887s" totalImagesPullingTime="352.249089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:00:07.990972006 +0000 UTC m=+844.851195532" watchObservedRunningTime="2026-07-17 22:00:07.992392887 +0000 UTC m=+844.852616388" Jul 17 22:00:18.989463 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:00:18.989432 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6fbf6969b9-dt6sj" Jul 17 22:05:20.993668 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:05:20.993626 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj"] Jul 17 22:05:20.994224 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:05:20.993870 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj" podUID="db1e2e3b-281f-4baa-bd34-c4d7c263e586" containerName="manager" containerID="cri-o://a1381f02d20c3a136ed3e5e13c12a5cc9816f30a71fed9754d4218372c954f5d" gracePeriod=10 Jul 17 22:05:24.356472 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:05:24.356449 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj" Jul 17 22:05:24.520933 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:05:24.520907 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cx75j\" (UniqueName: \"kubernetes.io/projected/db1e2e3b-281f-4baa-bd34-c4d7c263e586-kube-api-access-cx75j\") pod \"db1e2e3b-281f-4baa-bd34-c4d7c263e586\" (UID: \"db1e2e3b-281f-4baa-bd34-c4d7c263e586\") " Jul 17 22:05:24.521129 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:05:24.520988 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db1e2e3b-281f-4baa-bd34-c4d7c263e586-extensions-socket-volume\") pod \"db1e2e3b-281f-4baa-bd34-c4d7c263e586\" (UID: \"db1e2e3b-281f-4baa-bd34-c4d7c263e586\") " Jul 17 22:05:24.521316 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:05:24.521291 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1e2e3b-281f-4baa-bd34-c4d7c263e586-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "db1e2e3b-281f-4baa-bd34-c4d7c263e586" (UID: "db1e2e3b-281f-4baa-bd34-c4d7c263e586"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 17 22:05:24.523116 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:05:24.523093 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1e2e3b-281f-4baa-bd34-c4d7c263e586-kube-api-access-cx75j" (OuterVolumeSpecName: "kube-api-access-cx75j") pod "db1e2e3b-281f-4baa-bd34-c4d7c263e586" (UID: "db1e2e3b-281f-4baa-bd34-c4d7c263e586"). InnerVolumeSpecName "kube-api-access-cx75j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:05:24.622269 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:05:24.622240 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx75j\" (UniqueName: \"kubernetes.io/projected/db1e2e3b-281f-4baa-bd34-c4d7c263e586-kube-api-access-cx75j\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:05:24.622269 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:05:24.622265 2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db1e2e3b-281f-4baa-bd34-c4d7c263e586-extensions-socket-volume\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:05:24.632864 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:05:24.632835 2577 generic.go:356] "Generic (PLEG): container finished" podID="db1e2e3b-281f-4baa-bd34-c4d7c263e586" containerID="a1381f02d20c3a136ed3e5e13c12a5cc9816f30a71fed9754d4218372c954f5d" exitCode=0 Jul 17 22:05:24.632978 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:05:24.632883 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj" Jul 17 22:05:24.632978 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:05:24.632921 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj" event={"ID":"db1e2e3b-281f-4baa-bd34-c4d7c263e586","Type":"ContainerDied","Data":"a1381f02d20c3a136ed3e5e13c12a5cc9816f30a71fed9754d4218372c954f5d"} Jul 17 22:05:24.633049 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:05:24.633003 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj" event={"ID":"db1e2e3b-281f-4baa-bd34-c4d7c263e586","Type":"ContainerDied","Data":"28059f7a7a035cf9b002925e893f78a18c4427aac1d50f334a66fe2c11eeefab"} Jul 17 22:05:24.633049 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:05:24.633030 2577 scope.go:117] "RemoveContainer" containerID="a1381f02d20c3a136ed3e5e13c12a5cc9816f30a71fed9754d4218372c954f5d" Jul 17 22:05:24.647158 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:05:24.647139 2577 scope.go:117] "RemoveContainer" containerID="a1381f02d20c3a136ed3e5e13c12a5cc9816f30a71fed9754d4218372c954f5d" Jul 17 22:05:24.647437 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:05:24.647418 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1381f02d20c3a136ed3e5e13c12a5cc9816f30a71fed9754d4218372c954f5d\": container with ID starting with a1381f02d20c3a136ed3e5e13c12a5cc9816f30a71fed9754d4218372c954f5d not found: ID does not exist" containerID="a1381f02d20c3a136ed3e5e13c12a5cc9816f30a71fed9754d4218372c954f5d" Jul 17 22:05:24.647557 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:05:24.647446 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1381f02d20c3a136ed3e5e13c12a5cc9816f30a71fed9754d4218372c954f5d"} err="failed to get container status \"a1381f02d20c3a136ed3e5e13c12a5cc9816f30a71fed9754d4218372c954f5d\": rpc error: code = NotFound desc = could not find container \"a1381f02d20c3a136ed3e5e13c12a5cc9816f30a71fed9754d4218372c954f5d\": container with ID starting with a1381f02d20c3a136ed3e5e13c12a5cc9816f30a71fed9754d4218372c954f5d not found: ID does not exist" Jul 17 22:05:24.655758 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:05:24.655737 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj"] Jul 17 22:05:24.659505 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:05:24.659471 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-r72tj"] Jul 17 22:05:25.701322 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:05:25.701290 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db1e2e3b-281f-4baa-bd34-c4d7c263e586" path="/var/lib/kubelet/pods/db1e2e3b-281f-4baa-bd34-c4d7c263e586/volumes" Jul 17 22:06:22.112540 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:22.112499 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gnqr"] Jul 17 22:06:22.113119 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:22.113099 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db1e2e3b-281f-4baa-bd34-c4d7c263e586" containerName="manager" Jul 17 22:06:22.113202 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:22.113122 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="db1e2e3b-281f-4baa-bd34-c4d7c263e586" containerName="manager" Jul 17 22:06:22.113256 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:22.113220 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="db1e2e3b-281f-4baa-bd34-c4d7c263e586" containerName="manager" Jul 17 22:06:22.116365 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:22.116345 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gnqr" Jul 17 22:06:22.119014 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:22.118996 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-t2xkx\"" Jul 17 22:06:22.127704 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:22.127685 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gnqr"] Jul 17 22:06:22.143385 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:22.143360 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d76ef34f-96e7-4272-aff7-71cff43926f5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8gnqr\" (UID: \"d76ef34f-96e7-4272-aff7-71cff43926f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gnqr" Jul 17 22:06:22.143481 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:22.143464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxnqq\" (UniqueName: \"kubernetes.io/projected/d76ef34f-96e7-4272-aff7-71cff43926f5-kube-api-access-cxnqq\") pod \"kuadrant-operator-controller-manager-86878c769f-8gnqr\" (UID: \"d76ef34f-96e7-4272-aff7-71cff43926f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gnqr" Jul 17 22:06:22.243973 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:22.243941 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d76ef34f-96e7-4272-aff7-71cff43926f5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8gnqr\" (UID: \"d76ef34f-96e7-4272-aff7-71cff43926f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gnqr" Jul 17 22:06:22.244135 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:22.244025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxnqq\" (UniqueName: \"kubernetes.io/projected/d76ef34f-96e7-4272-aff7-71cff43926f5-kube-api-access-cxnqq\") pod \"kuadrant-operator-controller-manager-86878c769f-8gnqr\" (UID: \"d76ef34f-96e7-4272-aff7-71cff43926f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gnqr" Jul 17 22:06:22.244379 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:22.244359 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d76ef34f-96e7-4272-aff7-71cff43926f5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8gnqr\" (UID: \"d76ef34f-96e7-4272-aff7-71cff43926f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gnqr" Jul 17 22:06:22.259975 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:22.259945 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxnqq\" (UniqueName: \"kubernetes.io/projected/d76ef34f-96e7-4272-aff7-71cff43926f5-kube-api-access-cxnqq\") pod \"kuadrant-operator-controller-manager-86878c769f-8gnqr\" (UID: \"d76ef34f-96e7-4272-aff7-71cff43926f5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gnqr" Jul 17 22:06:22.427204 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:22.427117 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gnqr" Jul 17 22:06:22.579383 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:22.579358 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gnqr"] Jul 17 22:06:22.582207 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:06:22.582180 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd76ef34f_96e7_4272_aff7_71cff43926f5.slice/crio-18a8a2a127e217da39e9f432003161f375bc021ea5070997f163b01f342675c7 WatchSource:0}: Error finding container 18a8a2a127e217da39e9f432003161f375bc021ea5070997f163b01f342675c7: Status 404 returned error can't find the container with id 18a8a2a127e217da39e9f432003161f375bc021ea5070997f163b01f342675c7 Jul 17 22:06:22.947703 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:22.947666 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gnqr" event={"ID":"d76ef34f-96e7-4272-aff7-71cff43926f5","Type":"ContainerStarted","Data":"1cf7b46ae4554579812dcbf5c9b9ba29b62959339c58f34e2a0a94011f8e610b"} Jul 17 22:06:22.947911 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:22.947716 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gnqr" event={"ID":"d76ef34f-96e7-4272-aff7-71cff43926f5","Type":"ContainerStarted","Data":"18a8a2a127e217da39e9f432003161f375bc021ea5070997f163b01f342675c7"} Jul 17 22:06:22.947911 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:22.947746 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gnqr" Jul 17 22:06:22.970656 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:22.970606 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gnqr" podStartSLOduration=0.970590839 podStartE2EDuration="970.590839ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:06:22.969291199 +0000 UTC m=+1219.829514723" watchObservedRunningTime="2026-07-17 22:06:22.970590839 +0000 UTC m=+1219.830814341" Jul 17 22:06:33.954350 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:06:33.954323 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8gnqr" Jul 17 22:12:42.450621 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:42.450582 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv"] Jul 17 22:12:42.455026 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:42.455004 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:12:42.457471 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:42.457450 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7fb802f2-serving-cert\"" Jul 17 22:12:42.465283 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:42.465257 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv"] Jul 17 22:12:42.543900 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:42.543872 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhpc6\" (UniqueName: \"kubernetes.io/projected/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-kube-api-access-jhpc6\") pod \"maas-api-e2e-ait-7fb802f2-68df485674-f8pqv\" (UID: \"e94e71a0-be29-4647-9bba-72a1eb9bdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:12:42.544084 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:42.543916 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-openshift-service-ca\") pod \"maas-api-e2e-ait-7fb802f2-68df485674-f8pqv\" (UID: \"e94e71a0-be29-4647-9bba-72a1eb9bdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:12:42.544084 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:42.543990 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-maas-api-tls\") pod \"maas-api-e2e-ait-7fb802f2-68df485674-f8pqv\" (UID: \"e94e71a0-be29-4647-9bba-72a1eb9bdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:12:42.645189 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:42.645156 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-openshift-service-ca\") pod \"maas-api-e2e-ait-7fb802f2-68df485674-f8pqv\" (UID: \"e94e71a0-be29-4647-9bba-72a1eb9bdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:12:42.645382 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:42.645218 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-maas-api-tls\") pod \"maas-api-e2e-ait-7fb802f2-68df485674-f8pqv\" (UID: \"e94e71a0-be29-4647-9bba-72a1eb9bdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:12:42.645382 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:42.645267 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhpc6\" (UniqueName: \"kubernetes.io/projected/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-kube-api-access-jhpc6\") pod \"maas-api-e2e-ait-7fb802f2-68df485674-f8pqv\" (UID: \"e94e71a0-be29-4647-9bba-72a1eb9bdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:12:42.645548 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:12:42.645378 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-serving-cert: secret "maas-api-e2e-ait-7fb802f2-serving-cert" not found Jul 17 22:12:42.645548 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:12:42.645454 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-maas-api-tls podName:e94e71a0-be29-4647-9bba-72a1eb9bdbf4 nodeName:}" failed. No retries permitted until 2026-07-17 22:12:43.145437033 +0000 UTC m=+1600.005660513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-maas-api-tls") pod "maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" (UID: "e94e71a0-be29-4647-9bba-72a1eb9bdbf4") : secret "maas-api-e2e-ait-7fb802f2-serving-cert" not found Jul 17 22:12:42.645932 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:42.645910 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-openshift-service-ca\") pod \"maas-api-e2e-ait-7fb802f2-68df485674-f8pqv\" (UID: \"e94e71a0-be29-4647-9bba-72a1eb9bdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:12:42.655168 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:42.655147 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhpc6\" (UniqueName: \"kubernetes.io/projected/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-kube-api-access-jhpc6\") pod \"maas-api-e2e-ait-7fb802f2-68df485674-f8pqv\" (UID: \"e94e71a0-be29-4647-9bba-72a1eb9bdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:12:43.150672 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:43.150638 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-maas-api-tls\") pod \"maas-api-e2e-ait-7fb802f2-68df485674-f8pqv\" (UID: \"e94e71a0-be29-4647-9bba-72a1eb9bdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:12:43.153197 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:43.153174 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-maas-api-tls\") pod \"maas-api-e2e-ait-7fb802f2-68df485674-f8pqv\" (UID: \"e94e71a0-be29-4647-9bba-72a1eb9bdbf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:12:43.366758 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:43.366719 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:12:43.505657 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:43.505634 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv"] Jul 17 22:12:43.508018 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:12:43.507988 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94e71a0_be29_4647_9bba_72a1eb9bdbf4.slice/crio-d37772f4d879b852dacf92a24ba9c0e70cdfc76b36c99969c4b74db2add913b1 WatchSource:0}: Error finding container d37772f4d879b852dacf92a24ba9c0e70cdfc76b36c99969c4b74db2add913b1: Status 404 returned error can't find the container with id d37772f4d879b852dacf92a24ba9c0e70cdfc76b36c99969c4b74db2add913b1 Jul 17 22:12:43.509301 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:43.509281 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 22:12:43.936576 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:43.936534 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" event={"ID":"e94e71a0-be29-4647-9bba-72a1eb9bdbf4","Type":"ContainerStarted","Data":"d37772f4d879b852dacf92a24ba9c0e70cdfc76b36c99969c4b74db2add913b1"} Jul 17 22:12:44.943484 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:44.943450 2577 generic.go:356] "Generic (PLEG): container finished" podID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerID="ba9b75e84634f5d8699db48dc9ff1391debb6aa68d4d6509374544b6d2664e67" exitCode=1 Jul 17 22:12:44.943873 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:44.943547 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" event={"ID":"e94e71a0-be29-4647-9bba-72a1eb9bdbf4","Type":"ContainerDied","Data":"ba9b75e84634f5d8699db48dc9ff1391debb6aa68d4d6509374544b6d2664e67"} Jul 17 22:12:44.943873 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:44.943733 2577 scope.go:117] "RemoveContainer" containerID="ba9b75e84634f5d8699db48dc9ff1391debb6aa68d4d6509374544b6d2664e67" Jul 17 22:12:45.951025 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:45.950987 2577 generic.go:356] "Generic (PLEG): container finished" podID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerID="6c3c7eaac36b8b797e510f75dad78c81be890e46f3f88e4e2215fd0d2dd852ad" exitCode=1 Jul 17 22:12:45.951465 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:45.951076 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" event={"ID":"e94e71a0-be29-4647-9bba-72a1eb9bdbf4","Type":"ContainerDied","Data":"6c3c7eaac36b8b797e510f75dad78c81be890e46f3f88e4e2215fd0d2dd852ad"} Jul 17 22:12:45.951465 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:45.951140 2577 scope.go:117] "RemoveContainer" containerID="ba9b75e84634f5d8699db48dc9ff1391debb6aa68d4d6509374544b6d2664e67" Jul 17 22:12:45.951465 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:45.951374 2577 scope.go:117] "RemoveContainer" containerID="6c3c7eaac36b8b797e510f75dad78c81be890e46f3f88e4e2215fd0d2dd852ad" Jul 17 22:12:45.951638 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:12:45.951613 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:12:46.957852 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:46.957823 2577 scope.go:117] "RemoveContainer" containerID="6c3c7eaac36b8b797e510f75dad78c81be890e46f3f88e4e2215fd0d2dd852ad" Jul 17 22:12:46.958243 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:12:46.958040 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:12:48.367399 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:48.367359 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:12:48.367901 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:48.367883 2577 scope.go:117] "RemoveContainer" containerID="6c3c7eaac36b8b797e510f75dad78c81be890e46f3f88e4e2215fd0d2dd852ad" Jul 17 22:12:48.368108 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:12:48.368085 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:12:53.367653 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:53.367618 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:12:53.368032 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:12:53.368017 2577 scope.go:117] "RemoveContainer" containerID="6c3c7eaac36b8b797e510f75dad78c81be890e46f3f88e4e2215fd0d2dd852ad" Jul 17 22:12:53.368216 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:12:53.368195 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:13:06.694278 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:06.694248 2577 scope.go:117] "RemoveContainer" containerID="6c3c7eaac36b8b797e510f75dad78c81be890e46f3f88e4e2215fd0d2dd852ad" Jul 17 22:13:08.066995 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:08.066962 2577 generic.go:356] "Generic (PLEG): container finished" podID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerID="2a5a4a7c6bcfbea81e8788af2268bbe891e927a44118670ee42030e837330ce9" exitCode=1 Jul 17 22:13:08.067411 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:08.067015 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" event={"ID":"e94e71a0-be29-4647-9bba-72a1eb9bdbf4","Type":"ContainerDied","Data":"2a5a4a7c6bcfbea81e8788af2268bbe891e927a44118670ee42030e837330ce9"} Jul 17 22:13:08.067411 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:08.067063 2577 scope.go:117] "RemoveContainer" containerID="6c3c7eaac36b8b797e510f75dad78c81be890e46f3f88e4e2215fd0d2dd852ad" Jul 17 22:13:08.067411 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:08.067365 2577 scope.go:117] "RemoveContainer" containerID="2a5a4a7c6bcfbea81e8788af2268bbe891e927a44118670ee42030e837330ce9" Jul 17 22:13:08.067678 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:13:08.067647 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:13:08.367385 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:08.367308 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:13:09.075316 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:09.075292 2577 scope.go:117] "RemoveContainer" containerID="2a5a4a7c6bcfbea81e8788af2268bbe891e927a44118670ee42030e837330ce9" Jul 17 22:13:09.075727 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:13:09.075463 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:13:13.367624 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:13.367585 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:13:13.368122 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:13.368026 2577 scope.go:117] "RemoveContainer" containerID="2a5a4a7c6bcfbea81e8788af2268bbe891e927a44118670ee42030e837330ce9" Jul 17 22:13:13.368220 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:13:13.368199 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:13:25.694902 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:25.694868 2577 scope.go:117] "RemoveContainer" containerID="2a5a4a7c6bcfbea81e8788af2268bbe891e927a44118670ee42030e837330ce9" Jul 17 22:13:25.695311 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:13:25.695073 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:13:39.700323 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:39.700287 2577 scope.go:117] "RemoveContainer" containerID="2a5a4a7c6bcfbea81e8788af2268bbe891e927a44118670ee42030e837330ce9" Jul 17 22:13:40.230123 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:40.230088 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" event={"ID":"e94e71a0-be29-4647-9bba-72a1eb9bdbf4","Type":"ContainerStarted","Data":"793cc9b5e1efea357ca14fb96ab29d58f912b2d5240f4a04dc1801f07242e06a"} Jul 17 22:13:40.230306 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:40.230288 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:13:40.248526 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:40.247793 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podStartSLOduration=56.62251343 podStartE2EDuration="58.247774321s" totalImagesPullingTime="1.625260891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:12:42 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:13:40.246394555 +0000 UTC m=+1657.106618059" watchObservedRunningTime="2026-07-17 22:13:40.247774321 +0000 UTC m=+1657.107997824" Jul 17 22:13:41.236792 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:41.236758 2577 generic.go:356] "Generic (PLEG): container finished" podID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerID="793cc9b5e1efea357ca14fb96ab29d58f912b2d5240f4a04dc1801f07242e06a" exitCode=1 Jul 17 22:13:41.237242 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:41.236808 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" event={"ID":"e94e71a0-be29-4647-9bba-72a1eb9bdbf4","Type":"ContainerDied","Data":"793cc9b5e1efea357ca14fb96ab29d58f912b2d5240f4a04dc1801f07242e06a"} Jul 17 22:13:41.237242 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:41.236855 2577 scope.go:117] "RemoveContainer" containerID="2a5a4a7c6bcfbea81e8788af2268bbe891e927a44118670ee42030e837330ce9" Jul 17 22:13:41.237242 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:41.237056 2577 scope.go:117] "RemoveContainer" containerID="793cc9b5e1efea357ca14fb96ab29d58f912b2d5240f4a04dc1801f07242e06a" Jul 17 22:13:41.237362 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:13:41.237274 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:13:42.244201 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:42.244175 2577 scope.go:117] "RemoveContainer" containerID="793cc9b5e1efea357ca14fb96ab29d58f912b2d5240f4a04dc1801f07242e06a" Jul 17 22:13:42.244576 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:13:42.244351 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:13:43.367503 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:43.367472 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:13:43.367882 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:43.367867 2577 scope.go:117] "RemoveContainer" containerID="793cc9b5e1efea357ca14fb96ab29d58f912b2d5240f4a04dc1801f07242e06a" Jul 17 22:13:43.368063 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:13:43.368044 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:13:55.700153 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:13:55.700129 2577 scope.go:117] "RemoveContainer" containerID="793cc9b5e1efea357ca14fb96ab29d58f912b2d5240f4a04dc1801f07242e06a" Jul 17 22:13:55.700618 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:13:55.700314 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:14:10.694744 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:14:10.694714 2577 scope.go:117] "RemoveContainer" containerID="793cc9b5e1efea357ca14fb96ab29d58f912b2d5240f4a04dc1801f07242e06a" Jul 17 22:14:10.695133 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:14:10.694886 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:14:21.694741 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:14:21.694709 2577 scope.go:117] "RemoveContainer" containerID="793cc9b5e1efea357ca14fb96ab29d58f912b2d5240f4a04dc1801f07242e06a" Jul 17 22:14:22.466266 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:14:22.466230 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" event={"ID":"e94e71a0-be29-4647-9bba-72a1eb9bdbf4","Type":"ContainerStarted","Data":"99f9d3091488670a5a5e484f1ffdd43a4a609d555110bfc93a7ae179e657650a"} Jul 17 22:14:22.466480 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:14:22.466450 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:14:23.473560 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:14:23.473526 2577 generic.go:356] "Generic (PLEG): container finished" podID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerID="99f9d3091488670a5a5e484f1ffdd43a4a609d555110bfc93a7ae179e657650a" exitCode=1 Jul 17 22:14:23.473964 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:14:23.473599 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" event={"ID":"e94e71a0-be29-4647-9bba-72a1eb9bdbf4","Type":"ContainerDied","Data":"99f9d3091488670a5a5e484f1ffdd43a4a609d555110bfc93a7ae179e657650a"} Jul 17 22:14:23.473964 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:14:23.473664 2577 scope.go:117] "RemoveContainer" containerID="793cc9b5e1efea357ca14fb96ab29d58f912b2d5240f4a04dc1801f07242e06a" Jul 17 22:14:23.473964 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:14:23.473900 2577 scope.go:117] "RemoveContainer" containerID="99f9d3091488670a5a5e484f1ffdd43a4a609d555110bfc93a7ae179e657650a" Jul 17 22:14:23.474155 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:14:23.474132 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:14:24.480926 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:14:24.480859 2577 scope.go:117] "RemoveContainer" containerID="99f9d3091488670a5a5e484f1ffdd43a4a609d555110bfc93a7ae179e657650a" Jul 17 22:14:24.481281 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:14:24.481042 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:14:33.367536 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:14:33.367484 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:14:33.368008 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:14:33.367966 2577 scope.go:117] "RemoveContainer" containerID="99f9d3091488670a5a5e484f1ffdd43a4a609d555110bfc93a7ae179e657650a" Jul 17 22:14:33.368181 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:14:33.368158 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:14:47.694403 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:14:47.694369 2577 scope.go:117] "RemoveContainer" containerID="99f9d3091488670a5a5e484f1ffdd43a4a609d555110bfc93a7ae179e657650a" Jul 17 22:14:47.694942 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:14:47.694571 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:15:00.694164 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:00.694131 2577 scope.go:117] "RemoveContainer" containerID="99f9d3091488670a5a5e484f1ffdd43a4a609d555110bfc93a7ae179e657650a" Jul 17 22:15:00.694662 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:15:00.694300 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:15:12.694391 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:12.694358 2577 scope.go:117] "RemoveContainer" containerID="99f9d3091488670a5a5e484f1ffdd43a4a609d555110bfc93a7ae179e657650a" Jul 17 22:15:12.694800 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:15:12.694607 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:15:25.694484 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:25.694454 2577 scope.go:117] "RemoveContainer" containerID="99f9d3091488670a5a5e484f1ffdd43a4a609d555110bfc93a7ae179e657650a" Jul 17 22:15:25.694902 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:15:25.694674 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:15:36.694505 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:36.694467 2577 scope.go:117] "RemoveContainer" containerID="99f9d3091488670a5a5e484f1ffdd43a4a609d555110bfc93a7ae179e657650a" Jul 17 22:15:36.695034 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:15:36.694777 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:15:51.694484 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:51.694451 2577 scope.go:117] "RemoveContainer" containerID="99f9d3091488670a5a5e484f1ffdd43a4a609d555110bfc93a7ae179e657650a" Jul 17 22:15:52.933622 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:52.933545 2577 generic.go:356] "Generic (PLEG): container finished" podID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerID="367f35064c3fde1c8a76d972880c788c60e7888b2fec93baa1b957fa350ee6d1" exitCode=1 Jul 17 22:15:52.934005 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:52.933615 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" event={"ID":"e94e71a0-be29-4647-9bba-72a1eb9bdbf4","Type":"ContainerDied","Data":"367f35064c3fde1c8a76d972880c788c60e7888b2fec93baa1b957fa350ee6d1"} Jul 17 22:15:52.934005 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:52.933674 2577 scope.go:117] "RemoveContainer" containerID="99f9d3091488670a5a5e484f1ffdd43a4a609d555110bfc93a7ae179e657650a" Jul 17 22:15:52.934005 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:52.933972 2577 scope.go:117] "RemoveContainer" containerID="367f35064c3fde1c8a76d972880c788c60e7888b2fec93baa1b957fa350ee6d1" Jul 17 22:15:52.934218 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:15:52.934199 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-7fb802f2-68df485674-f8pqv_odh-ai-gateway-infra(e94e71a0-be29-4647-9bba-72a1eb9bdbf4)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" Jul 17 22:15:53.029185 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:53.029153 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv"] Jul 17 22:15:53.083980 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:15:53.083945 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-serving-cert: secret "maas-api-e2e-ait-7fb802f2-serving-cert" not found Jul 17 22:15:53.084165 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:15:53.084014 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-maas-api-tls podName:e94e71a0-be29-4647-9bba-72a1eb9bdbf4 nodeName:}" failed. No retries permitted until 2026-07-17 22:15:53.583995259 +0000 UTC m=+1790.444218740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-maas-api-tls") pod "maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" (UID: "e94e71a0-be29-4647-9bba-72a1eb9bdbf4") : secret "maas-api-e2e-ait-7fb802f2-serving-cert" not found Jul 17 22:15:53.366983 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:53.366939 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:15:53.589043 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:15:53.589003 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-serving-cert: secret "maas-api-e2e-ait-7fb802f2-serving-cert" not found Jul 17 22:15:53.589215 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:15:53.589082 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-maas-api-tls podName:e94e71a0-be29-4647-9bba-72a1eb9bdbf4 nodeName:}" failed. No retries permitted until 2026-07-17 22:15:54.589064078 +0000 UTC m=+1791.449287565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-maas-api-tls") pod "maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" (UID: "e94e71a0-be29-4647-9bba-72a1eb9bdbf4") : secret "maas-api-e2e-ait-7fb802f2-serving-cert" not found Jul 17 22:15:54.090861 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:54.090836 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:15:54.194914 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:54.194823 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-maas-api-tls\") pod \"e94e71a0-be29-4647-9bba-72a1eb9bdbf4\" (UID: \"e94e71a0-be29-4647-9bba-72a1eb9bdbf4\") " Jul 17 22:15:54.194914 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:54.194882 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhpc6\" (UniqueName: \"kubernetes.io/projected/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-kube-api-access-jhpc6\") pod \"e94e71a0-be29-4647-9bba-72a1eb9bdbf4\" (UID: \"e94e71a0-be29-4647-9bba-72a1eb9bdbf4\") " Jul 17 22:15:54.194914 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:54.194901 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-openshift-service-ca\") pod \"e94e71a0-be29-4647-9bba-72a1eb9bdbf4\" (UID: \"e94e71a0-be29-4647-9bba-72a1eb9bdbf4\") " Jul 17 22:15:54.195306 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:54.195283 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e94e71a0-be29-4647-9bba-72a1eb9bdbf4" (UID: "e94e71a0-be29-4647-9bba-72a1eb9bdbf4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:15:54.197256 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:54.197234 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e94e71a0-be29-4647-9bba-72a1eb9bdbf4" (UID: "e94e71a0-be29-4647-9bba-72a1eb9bdbf4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:15:54.197365 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:54.197264 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-kube-api-access-jhpc6" (OuterVolumeSpecName: "kube-api-access-jhpc6") pod "e94e71a0-be29-4647-9bba-72a1eb9bdbf4" (UID: "e94e71a0-be29-4647-9bba-72a1eb9bdbf4"). InnerVolumeSpecName "kube-api-access-jhpc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:15:54.296577 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:54.296537 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-maas-api-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:15:54.296577 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:54.296574 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhpc6\" (UniqueName: \"kubernetes.io/projected/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-kube-api-access-jhpc6\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:15:54.296577 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:54.296584 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e94e71a0-be29-4647-9bba-72a1eb9bdbf4-openshift-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:15:54.946615 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:54.946586 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" Jul 17 22:15:54.946790 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:54.946627 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv" event={"ID":"e94e71a0-be29-4647-9bba-72a1eb9bdbf4","Type":"ContainerDied","Data":"d37772f4d879b852dacf92a24ba9c0e70cdfc76b36c99969c4b74db2add913b1"} Jul 17 22:15:54.946790 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:54.946692 2577 scope.go:117] "RemoveContainer" containerID="367f35064c3fde1c8a76d972880c788c60e7888b2fec93baa1b957fa350ee6d1" Jul 17 22:15:54.968626 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:54.968594 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv"] Jul 17 22:15:54.971999 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:54.971978 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7fb802f2-68df485674-f8pqv"] Jul 17 22:15:55.700981 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:15:55.700953 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" path="/var/lib/kubelet/pods/e94e71a0-be29-4647-9bba-72a1eb9bdbf4/volumes" Jul 17 22:16:01.255144 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.255102 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps"] Jul 17 22:16:01.255799 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.255782 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerName="maas-api" Jul 17 22:16:01.255870 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.255802 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerName="maas-api" Jul 17 22:16:01.255870 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.255824 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerName="maas-api" Jul 17 22:16:01.255870 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.255832 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerName="maas-api" Jul 17 22:16:01.255870 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.255864 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerName="maas-api" Jul 17 22:16:01.256041 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.255873 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerName="maas-api" Jul 17 22:16:01.256041 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.255892 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerName="maas-api" Jul 17 22:16:01.256041 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.255902 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerName="maas-api" Jul 17 22:16:01.256041 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.255988 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerName="maas-api" Jul 17 22:16:01.256041 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.255997 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerName="maas-api" Jul 17 22:16:01.256041 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.256008 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerName="maas-api" Jul 17 22:16:01.256041 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.256018 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerName="maas-api" Jul 17 22:16:01.256041 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.256027 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerName="maas-api" Jul 17 22:16:01.259567 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.259547 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" Jul 17 22:16:01.262717 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.262698 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-09e18326-serving-cert\"" Jul 17 22:16:01.272890 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.272855 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps"] Jul 17 22:16:01.365408 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.365318 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70-maas-api-tls\") pod \"maas-api-e2e-migrate-09e18326-989b4fd-bn5ps\" (UID: \"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" Jul 17 22:16:01.365408 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.365350 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kzd8\" (UniqueName: \"kubernetes.io/projected/0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70-kube-api-access-4kzd8\") pod \"maas-api-e2e-migrate-09e18326-989b4fd-bn5ps\" (UID: \"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" Jul 17 22:16:01.365638 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.365459 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70-openshift-service-ca\") pod \"maas-api-e2e-migrate-09e18326-989b4fd-bn5ps\" (UID: \"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" Jul 17 22:16:01.466101 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.466065 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70-maas-api-tls\") pod \"maas-api-e2e-migrate-09e18326-989b4fd-bn5ps\" (UID: \"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" Jul 17 22:16:01.466101 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.466099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kzd8\" (UniqueName: \"kubernetes.io/projected/0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70-kube-api-access-4kzd8\") pod \"maas-api-e2e-migrate-09e18326-989b4fd-bn5ps\" (UID: \"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" Jul 17 22:16:01.466320 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.466180 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70-openshift-service-ca\") pod \"maas-api-e2e-migrate-09e18326-989b4fd-bn5ps\" (UID: \"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" Jul 17 22:16:01.466941 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.466869 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70-openshift-service-ca\") pod \"maas-api-e2e-migrate-09e18326-989b4fd-bn5ps\" (UID: \"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" Jul 17 22:16:01.468770 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.468745 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70-maas-api-tls\") pod \"maas-api-e2e-migrate-09e18326-989b4fd-bn5ps\" (UID: \"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" Jul 17 22:16:01.474713 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.474689 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kzd8\" (UniqueName: \"kubernetes.io/projected/0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70-kube-api-access-4kzd8\") pod \"maas-api-e2e-migrate-09e18326-989b4fd-bn5ps\" (UID: \"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" Jul 17 22:16:01.573049 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.573015 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" Jul 17 22:16:01.722993 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.722971 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps"] Jul 17 22:16:01.725378 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:16:01.725334 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c2abcd7_1ef6_4826_ae84_5cf9bb19fb70.slice/crio-1aeba1bf0e25317b9f0b75261ccf272d073c335c1a5674e7f2bb1e320f370fb8 WatchSource:0}: Error finding container 1aeba1bf0e25317b9f0b75261ccf272d073c335c1a5674e7f2bb1e320f370fb8: Status 404 returned error can't find the container with id 1aeba1bf0e25317b9f0b75261ccf272d073c335c1a5674e7f2bb1e320f370fb8 Jul 17 22:16:01.983663 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:01.983625 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" event={"ID":"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70","Type":"ContainerStarted","Data":"1aeba1bf0e25317b9f0b75261ccf272d073c335c1a5674e7f2bb1e320f370fb8"} Jul 17 22:16:02.990933 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:02.990898 2577 generic.go:356] "Generic (PLEG): container finished" podID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" containerID="31561cefeb5c89469107fac2fea1f7b2122b5152b0f22a4a1cdc0ba3e2e8522a" exitCode=1 Jul 17 22:16:02.991325 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:02.990985 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" event={"ID":"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70","Type":"ContainerDied","Data":"31561cefeb5c89469107fac2fea1f7b2122b5152b0f22a4a1cdc0ba3e2e8522a"} Jul 17 22:16:02.991325 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:02.991203 2577 scope.go:117] "RemoveContainer" containerID="31561cefeb5c89469107fac2fea1f7b2122b5152b0f22a4a1cdc0ba3e2e8522a" Jul 17 22:16:04.004947 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:04.004845 2577 generic.go:356] "Generic (PLEG): container finished" podID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" containerID="3addf1ea2ef5acc2ed0623023fa1b1c423f7b9363a3408ef71553e4843db49b2" exitCode=1 Jul 17 22:16:04.005402 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:04.004981 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" event={"ID":"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70","Type":"ContainerDied","Data":"3addf1ea2ef5acc2ed0623023fa1b1c423f7b9363a3408ef71553e4843db49b2"} Jul 17 22:16:04.005402 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:04.005090 2577 scope.go:117] "RemoveContainer" containerID="31561cefeb5c89469107fac2fea1f7b2122b5152b0f22a4a1cdc0ba3e2e8522a" Jul 17 22:16:04.005795 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:04.005772 2577 scope.go:117] "RemoveContainer" containerID="3addf1ea2ef5acc2ed0623023fa1b1c423f7b9363a3408ef71553e4843db49b2" Jul 17 22:16:04.006605 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:16:04.006102 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-09e18326-989b4fd-bn5ps_odh-ai-gateway-infra(0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" Jul 17 22:16:05.012484 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:05.012457 2577 scope.go:117] "RemoveContainer" containerID="3addf1ea2ef5acc2ed0623023fa1b1c423f7b9363a3408ef71553e4843db49b2" Jul 17 22:16:05.012910 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:16:05.012672 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-09e18326-989b4fd-bn5ps_odh-ai-gateway-infra(0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" Jul 17 22:16:06.573579 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:06.573548 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" Jul 17 22:16:06.574020 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:06.573998 2577 scope.go:117] "RemoveContainer" containerID="3addf1ea2ef5acc2ed0623023fa1b1c423f7b9363a3408ef71553e4843db49b2" Jul 17 22:16:06.574226 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:16:06.574205 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-09e18326-989b4fd-bn5ps_odh-ai-gateway-infra(0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" Jul 17 22:16:11.573565 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:11.573454 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" Jul 17 22:16:11.574068 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:11.573955 2577 scope.go:117] "RemoveContainer" containerID="3addf1ea2ef5acc2ed0623023fa1b1c423f7b9363a3408ef71553e4843db49b2" Jul 17 22:16:11.574205 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:16:11.574181 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-migrate-09e18326-989b4fd-bn5ps_odh-ai-gateway-infra(0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" Jul 17 22:16:23.696898 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:23.696864 2577 scope.go:117] "RemoveContainer" containerID="3addf1ea2ef5acc2ed0623023fa1b1c423f7b9363a3408ef71553e4843db49b2" Jul 17 22:16:24.110850 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:24.110815 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" event={"ID":"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70","Type":"ContainerStarted","Data":"1800775c7af3e4e6bb257e3b64fffbebbe928c7bbc626ba102ccfbd9acf25305"} Jul 17 22:16:24.111051 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:24.111022 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" Jul 17 22:16:24.129550 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:24.129485 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" podStartSLOduration=22.01196308 podStartE2EDuration="23.129468883s" totalImagesPullingTime="1.117505803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:16:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:16:24.128408645 +0000 UTC m=+1820.988632147" watchObservedRunningTime="2026-07-17 22:16:24.129468883 +0000 UTC m=+1820.989692384" Jul 17 22:16:25.117424 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:25.117390 2577 generic.go:356] "Generic (PLEG): container finished" podID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" containerID="1800775c7af3e4e6bb257e3b64fffbebbe928c7bbc626ba102ccfbd9acf25305" exitCode=1 Jul 17 22:16:25.117940 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:25.117481 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" event={"ID":"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70","Type":"ContainerDied","Data":"1800775c7af3e4e6bb257e3b64fffbebbe928c7bbc626ba102ccfbd9acf25305"} Jul 17 22:16:25.117940 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:25.117551 2577 scope.go:117] "RemoveContainer" containerID="3addf1ea2ef5acc2ed0623023fa1b1c423f7b9363a3408ef71553e4843db49b2" Jul 17 22:16:25.117940 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:25.117763 2577 scope.go:117] "RemoveContainer" containerID="1800775c7af3e4e6bb257e3b64fffbebbe928c7bbc626ba102ccfbd9acf25305" Jul 17 22:16:25.118115 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:16:25.118014 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-09e18326-989b4fd-bn5ps_odh-ai-gateway-infra(0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" Jul 17 22:16:26.124671 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:26.124645 2577 scope.go:117] "RemoveContainer" containerID="1800775c7af3e4e6bb257e3b64fffbebbe928c7bbc626ba102ccfbd9acf25305" Jul 17 22:16:26.125051 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:16:26.124825 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-09e18326-989b4fd-bn5ps_odh-ai-gateway-infra(0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" Jul 17 22:16:31.574175 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:31.574141 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" Jul 17 22:16:31.574587 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:31.574559 2577 scope.go:117] "RemoveContainer" containerID="1800775c7af3e4e6bb257e3b64fffbebbe928c7bbc626ba102ccfbd9acf25305" Jul 17 22:16:31.574764 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:16:31.574745 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-migrate-09e18326-989b4fd-bn5ps_odh-ai-gateway-infra(0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" Jul 17 22:16:46.694213 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:46.694174 2577 scope.go:117] "RemoveContainer" containerID="1800775c7af3e4e6bb257e3b64fffbebbe928c7bbc626ba102ccfbd9acf25305" Jul 17 22:16:47.234022 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:47.233985 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" event={"ID":"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70","Type":"ContainerStarted","Data":"6502c6291be2260c4cb6d35b6474455da521e5d65d1b9318d0fddc2a749d7165"} Jul 17 22:16:47.234215 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:47.234198 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" Jul 17 22:16:48.241308 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:48.241273 2577 generic.go:356] "Generic (PLEG): container finished" podID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" containerID="6502c6291be2260c4cb6d35b6474455da521e5d65d1b9318d0fddc2a749d7165" exitCode=1 Jul 17 22:16:48.241742 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:48.241334 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" event={"ID":"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70","Type":"ContainerDied","Data":"6502c6291be2260c4cb6d35b6474455da521e5d65d1b9318d0fddc2a749d7165"} Jul 17 22:16:48.241742 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:48.241382 2577 scope.go:117] "RemoveContainer" containerID="1800775c7af3e4e6bb257e3b64fffbebbe928c7bbc626ba102ccfbd9acf25305" Jul 17 22:16:48.241742 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:48.241591 2577 scope.go:117] "RemoveContainer" containerID="6502c6291be2260c4cb6d35b6474455da521e5d65d1b9318d0fddc2a749d7165" Jul 17 22:16:48.241867 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:16:48.241795 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-09e18326-989b4fd-bn5ps_odh-ai-gateway-infra(0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" Jul 17 22:16:49.248283 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:49.248256 2577 scope.go:117] "RemoveContainer" containerID="6502c6291be2260c4cb6d35b6474455da521e5d65d1b9318d0fddc2a749d7165" Jul 17 22:16:49.248699 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:16:49.248430 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-09e18326-989b4fd-bn5ps_odh-ai-gateway-infra(0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" Jul 17 22:16:51.573268 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:51.573231 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" Jul 17 22:16:51.573717 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:16:51.573701 2577 scope.go:117] "RemoveContainer" containerID="6502c6291be2260c4cb6d35b6474455da521e5d65d1b9318d0fddc2a749d7165" Jul 17 22:16:51.573895 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:16:51.573874 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-09e18326-989b4fd-bn5ps_odh-ai-gateway-infra(0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" Jul 17 22:17:02.694755 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:02.694723 2577 scope.go:117] "RemoveContainer" containerID="6502c6291be2260c4cb6d35b6474455da521e5d65d1b9318d0fddc2a749d7165" Jul 17 22:17:02.695245 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:17:02.694903 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-migrate-09e18326-989b4fd-bn5ps_odh-ai-gateway-infra(0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70)\"" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" Jul 17 22:17:11.547251 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:11.547210 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps"] Jul 17 22:17:11.694243 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:11.694216 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" Jul 17 22:17:11.831693 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:11.831598 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kzd8\" (UniqueName: \"kubernetes.io/projected/0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70-kube-api-access-4kzd8\") pod \"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70\" (UID: \"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70\") " Jul 17 22:17:11.831693 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:11.831646 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70-maas-api-tls\") pod \"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70\" (UID: \"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70\") " Jul 17 22:17:11.831693 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:11.831673 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70-openshift-service-ca\") pod \"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70\" (UID: \"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70\") " Jul 17 22:17:11.832056 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:11.832016 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" (UID: "0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:17:11.832140 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:11.832105 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70-openshift-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:17:11.834013 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:11.833993 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" (UID: "0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:17:11.834152 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:11.834019 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70-kube-api-access-4kzd8" (OuterVolumeSpecName: "kube-api-access-4kzd8") pod "0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" (UID: "0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70"). InnerVolumeSpecName "kube-api-access-4kzd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:17:11.932944 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:11.932912 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kzd8\" (UniqueName: \"kubernetes.io/projected/0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70-kube-api-access-4kzd8\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:17:11.932944 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:11.932941 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70-maas-api-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:17:12.370719 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:12.370687 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" Jul 17 22:17:12.370914 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:12.370720 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps" event={"ID":"0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70","Type":"ContainerDied","Data":"1aeba1bf0e25317b9f0b75261ccf272d073c335c1a5674e7f2bb1e320f370fb8"} Jul 17 22:17:12.370914 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:12.370782 2577 scope.go:117] "RemoveContainer" containerID="6502c6291be2260c4cb6d35b6474455da521e5d65d1b9318d0fddc2a749d7165" Jul 17 22:17:12.396270 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:12.396248 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps"] Jul 17 22:17:12.400373 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:12.400348 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-09e18326-989b4fd-bn5ps"] Jul 17 22:17:13.700267 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:13.700231 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" path="/var/lib/kubelet/pods/0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70/volumes" Jul 17 22:17:19.328890 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.328858 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc"] Jul 17 22:17:19.329431 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.329274 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" containerName="maas-api" Jul 17 22:17:19.329431 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.329284 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" containerName="maas-api" Jul 17 22:17:19.329431 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.329297 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" containerName="maas-api" Jul 17 22:17:19.329431 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.329302 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" containerName="maas-api" Jul 17 22:17:19.329431 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.329316 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerName="maas-api" Jul 17 22:17:19.329431 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.329322 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerName="maas-api" Jul 17 22:17:19.329431 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.329330 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" containerName="maas-api" Jul 17 22:17:19.329431 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.329335 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" containerName="maas-api" Jul 17 22:17:19.329431 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.329343 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerName="maas-api" Jul 17 22:17:19.329431 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.329348 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerName="maas-api" Jul 17 22:17:19.329431 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.329405 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" containerName="maas-api" Jul 17 22:17:19.329431 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.329411 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" containerName="maas-api" Jul 17 22:17:19.329431 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.329418 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" containerName="maas-api" Jul 17 22:17:19.329431 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.329423 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e94e71a0-be29-4647-9bba-72a1eb9bdbf4" containerName="maas-api" Jul 17 22:17:19.332682 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.332661 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:17:19.335053 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.335027 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-45495ac6-serving-cert\"" Jul 17 22:17:19.346743 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.346708 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc"] Jul 17 22:17:19.503632 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.503598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8ljr\" (UniqueName: \"kubernetes.io/projected/42b8b1a2-2552-46d0-985b-f7bc337a15bf-kube-api-access-b8ljr\") pod \"maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc\" (UID: \"42b8b1a2-2552-46d0-985b-f7bc337a15bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:17:19.503827 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.503661 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42b8b1a2-2552-46d0-985b-f7bc337a15bf-maas-api-tls\") pod \"maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc\" (UID: \"42b8b1a2-2552-46d0-985b-f7bc337a15bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:17:19.503827 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.503693 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b8b1a2-2552-46d0-985b-f7bc337a15bf-openshift-service-ca\") pod \"maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc\" (UID: \"42b8b1a2-2552-46d0-985b-f7bc337a15bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:17:19.604634 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.604531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8ljr\" (UniqueName: \"kubernetes.io/projected/42b8b1a2-2552-46d0-985b-f7bc337a15bf-kube-api-access-b8ljr\") pod \"maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc\" (UID: \"42b8b1a2-2552-46d0-985b-f7bc337a15bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:17:19.604634 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.604594 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42b8b1a2-2552-46d0-985b-f7bc337a15bf-maas-api-tls\") pod \"maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc\" (UID: \"42b8b1a2-2552-46d0-985b-f7bc337a15bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:17:19.604634 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.604626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b8b1a2-2552-46d0-985b-f7bc337a15bf-openshift-service-ca\") pod \"maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc\" (UID: \"42b8b1a2-2552-46d0-985b-f7bc337a15bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:17:19.605380 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.605273 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b8b1a2-2552-46d0-985b-f7bc337a15bf-openshift-service-ca\") pod \"maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc\" (UID: \"42b8b1a2-2552-46d0-985b-f7bc337a15bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:17:19.607225 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.607203 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42b8b1a2-2552-46d0-985b-f7bc337a15bf-maas-api-tls\") pod \"maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc\" (UID: \"42b8b1a2-2552-46d0-985b-f7bc337a15bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:17:19.613005 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.612975 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8ljr\" (UniqueName: \"kubernetes.io/projected/42b8b1a2-2552-46d0-985b-f7bc337a15bf-kube-api-access-b8ljr\") pod \"maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc\" (UID: \"42b8b1a2-2552-46d0-985b-f7bc337a15bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:17:19.645989 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.645954 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:17:19.797298 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:19.797272 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc"] Jul 17 22:17:19.799004 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:17:19.798978 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b8b1a2_2552_46d0_985b_f7bc337a15bf.slice/crio-0e49ecef1c51be8cdba7d92a432cccc8f68533a000846191bf2f7894d986ab96 WatchSource:0}: Error finding container 0e49ecef1c51be8cdba7d92a432cccc8f68533a000846191bf2f7894d986ab96: Status 404 returned error can't find the container with id 0e49ecef1c51be8cdba7d92a432cccc8f68533a000846191bf2f7894d986ab96 Jul 17 22:17:20.417397 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:20.417361 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" event={"ID":"42b8b1a2-2552-46d0-985b-f7bc337a15bf","Type":"ContainerStarted","Data":"66024013f9bb5dd7163514e877e1e88f6634cde8929f86a37033e719c8e5e670"} Jul 17 22:17:20.417844 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:20.417431 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:17:20.417844 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:20.417452 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" event={"ID":"42b8b1a2-2552-46d0-985b-f7bc337a15bf","Type":"ContainerStarted","Data":"0e49ecef1c51be8cdba7d92a432cccc8f68533a000846191bf2f7894d986ab96"} Jul 17 22:17:20.434600 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:20.434547 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podStartSLOduration=0.963166905 podStartE2EDuration="1.434530226s" totalImagesPullingTime="471.363321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:17:20.433553462 +0000 UTC m=+1877.293776983" watchObservedRunningTime="2026-07-17 22:17:20.434530226 +0000 UTC m=+1877.294753727" Jul 17 22:17:21.425074 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:21.424966 2577 generic.go:356] "Generic (PLEG): container finished" podID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerID="66024013f9bb5dd7163514e877e1e88f6634cde8929f86a37033e719c8e5e670" exitCode=1 Jul 17 22:17:21.425074 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:21.425056 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" event={"ID":"42b8b1a2-2552-46d0-985b-f7bc337a15bf","Type":"ContainerDied","Data":"66024013f9bb5dd7163514e877e1e88f6634cde8929f86a37033e719c8e5e670"} Jul 17 22:17:21.425560 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:21.425323 2577 scope.go:117] "RemoveContainer" containerID="66024013f9bb5dd7163514e877e1e88f6634cde8929f86a37033e719c8e5e670" Jul 17 22:17:22.432018 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:22.431925 2577 generic.go:356] "Generic (PLEG): container finished" podID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerID="7158e840e078f2ab3c05793dea3f2d7e0f6e24397dd5ef37ff0c26d23bbf7c8c" exitCode=1 Jul 17 22:17:22.432412 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:22.432014 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" event={"ID":"42b8b1a2-2552-46d0-985b-f7bc337a15bf","Type":"ContainerDied","Data":"7158e840e078f2ab3c05793dea3f2d7e0f6e24397dd5ef37ff0c26d23bbf7c8c"} Jul 17 22:17:22.432412 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:22.432104 2577 scope.go:117] "RemoveContainer" containerID="66024013f9bb5dd7163514e877e1e88f6634cde8929f86a37033e719c8e5e670" Jul 17 22:17:22.432412 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:22.432317 2577 scope.go:117] "RemoveContainer" containerID="7158e840e078f2ab3c05793dea3f2d7e0f6e24397dd5ef37ff0c26d23bbf7c8c" Jul 17 22:17:22.432654 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:17:22.432554 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:17:23.439418 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:23.439374 2577 scope.go:117] "RemoveContainer" containerID="7158e840e078f2ab3c05793dea3f2d7e0f6e24397dd5ef37ff0c26d23bbf7c8c" Jul 17 22:17:23.439817 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:17:23.439571 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:17:25.418255 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:25.418177 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:17:25.418629 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:25.418604 2577 scope.go:117] "RemoveContainer" containerID="7158e840e078f2ab3c05793dea3f2d7e0f6e24397dd5ef37ff0c26d23bbf7c8c" Jul 17 22:17:25.418795 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:17:25.418775 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:17:29.646374 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:29.646338 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:17:29.646836 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:29.646782 2577 scope.go:117] "RemoveContainer" containerID="7158e840e078f2ab3c05793dea3f2d7e0f6e24397dd5ef37ff0c26d23bbf7c8c" Jul 17 22:17:29.646967 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:17:29.646947 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:17:42.694386 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:42.694353 2577 scope.go:117] "RemoveContainer" containerID="7158e840e078f2ab3c05793dea3f2d7e0f6e24397dd5ef37ff0c26d23bbf7c8c" Jul 17 22:17:43.548994 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:43.548966 2577 generic.go:356] "Generic (PLEG): container finished" podID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerID="9675607e5e27656e794b065629d2c72ee11fda945686958ddb2bc9a821007dae" exitCode=1 Jul 17 22:17:43.549158 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:43.549046 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" event={"ID":"42b8b1a2-2552-46d0-985b-f7bc337a15bf","Type":"ContainerDied","Data":"9675607e5e27656e794b065629d2c72ee11fda945686958ddb2bc9a821007dae"} Jul 17 22:17:43.549158 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:43.549114 2577 scope.go:117] "RemoveContainer" containerID="7158e840e078f2ab3c05793dea3f2d7e0f6e24397dd5ef37ff0c26d23bbf7c8c" Jul 17 22:17:43.549511 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:43.549464 2577 scope.go:117] "RemoveContainer" containerID="9675607e5e27656e794b065629d2c72ee11fda945686958ddb2bc9a821007dae" Jul 17 22:17:43.549740 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:17:43.549718 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:17:45.417985 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:45.417950 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:17:45.418471 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:45.418402 2577 scope.go:117] "RemoveContainer" containerID="9675607e5e27656e794b065629d2c72ee11fda945686958ddb2bc9a821007dae" Jul 17 22:17:45.418687 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:17:45.418665 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:17:49.646626 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:49.646595 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:17:49.647085 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:17:49.646980 2577 scope.go:117] "RemoveContainer" containerID="9675607e5e27656e794b065629d2c72ee11fda945686958ddb2bc9a821007dae" Jul 17 22:17:49.647163 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:17:49.647142 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:18:00.695319 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:00.695282 2577 scope.go:117] "RemoveContainer" containerID="9675607e5e27656e794b065629d2c72ee11fda945686958ddb2bc9a821007dae" Jul 17 22:18:00.695799 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:18:00.695537 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:18:14.694018 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:14.693990 2577 scope.go:117] "RemoveContainer" containerID="9675607e5e27656e794b065629d2c72ee11fda945686958ddb2bc9a821007dae" Jul 17 22:18:14.694403 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:14.694289 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 22:18:15.711335 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:15.711306 2577 generic.go:356] "Generic (PLEG): container finished" podID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerID="82b623597123d4002b6078efb5861a02e5931854c62939422bf4eeb2d3066bcb" exitCode=1 Jul 17 22:18:15.711753 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:15.711354 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" event={"ID":"42b8b1a2-2552-46d0-985b-f7bc337a15bf","Type":"ContainerDied","Data":"82b623597123d4002b6078efb5861a02e5931854c62939422bf4eeb2d3066bcb"} Jul 17 22:18:15.711753 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:15.711395 2577 scope.go:117] "RemoveContainer" containerID="9675607e5e27656e794b065629d2c72ee11fda945686958ddb2bc9a821007dae" Jul 17 22:18:15.711753 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:15.711710 2577 scope.go:117] "RemoveContainer" containerID="82b623597123d4002b6078efb5861a02e5931854c62939422bf4eeb2d3066bcb" Jul 17 22:18:15.711895 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:18:15.711876 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:18:19.647061 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:19.647026 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:18:19.647457 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:19.647406 2577 scope.go:117] "RemoveContainer" containerID="82b623597123d4002b6078efb5861a02e5931854c62939422bf4eeb2d3066bcb" Jul 17 22:18:19.647633 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:18:19.647611 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:18:20.417605 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:20.417560 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:18:20.417943 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:20.417930 2577 scope.go:117] "RemoveContainer" containerID="82b623597123d4002b6078efb5861a02e5931854c62939422bf4eeb2d3066bcb" Jul 17 22:18:20.418119 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:18:20.418100 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:18:34.694690 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:34.694657 2577 scope.go:117] "RemoveContainer" containerID="82b623597123d4002b6078efb5861a02e5931854c62939422bf4eeb2d3066bcb" Jul 17 22:18:34.695169 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:18:34.694831 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:18:45.694776 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:45.694746 2577 scope.go:117] "RemoveContainer" containerID="82b623597123d4002b6078efb5861a02e5931854c62939422bf4eeb2d3066bcb" Jul 17 22:18:45.695295 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:18:45.694932 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:18:57.694545 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:57.694442 2577 scope.go:117] "RemoveContainer" containerID="82b623597123d4002b6078efb5861a02e5931854c62939422bf4eeb2d3066bcb" Jul 17 22:18:58.934456 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:58.934425 2577 generic.go:356] "Generic (PLEG): container finished" podID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerID="78d24cc9a9b1dd62d9c0736542643a0c3c011ea91214340fb59264c8068847aa" exitCode=1 Jul 17 22:18:58.934887 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:58.934507 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" event={"ID":"42b8b1a2-2552-46d0-985b-f7bc337a15bf","Type":"ContainerDied","Data":"78d24cc9a9b1dd62d9c0736542643a0c3c011ea91214340fb59264c8068847aa"} Jul 17 22:18:58.934887 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:58.934571 2577 scope.go:117] "RemoveContainer" containerID="82b623597123d4002b6078efb5861a02e5931854c62939422bf4eeb2d3066bcb" Jul 17 22:18:58.934887 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:58.934872 2577 scope.go:117] "RemoveContainer" containerID="78d24cc9a9b1dd62d9c0736542643a0c3c011ea91214340fb59264c8068847aa" Jul 17 22:18:58.935093 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:18:58.935067 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:18:59.647084 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:59.647049 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:18:59.941679 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:18:59.941655 2577 scope.go:117] "RemoveContainer" containerID="78d24cc9a9b1dd62d9c0736542643a0c3c011ea91214340fb59264c8068847aa" Jul 17 22:18:59.942060 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:18:59.941834 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:19:00.418214 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:19:00.418113 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:19:00.947137 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:19:00.947109 2577 scope.go:117] "RemoveContainer" containerID="78d24cc9a9b1dd62d9c0736542643a0c3c011ea91214340fb59264c8068847aa" Jul 17 22:19:00.947547 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:19:00.947280 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:19:13.697246 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:19:13.697198 2577 scope.go:117] "RemoveContainer" containerID="78d24cc9a9b1dd62d9c0736542643a0c3c011ea91214340fb59264c8068847aa" Jul 17 22:19:13.697744 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:19:13.697415 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:19:27.695017 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:19:27.694977 2577 scope.go:117] "RemoveContainer" containerID="78d24cc9a9b1dd62d9c0736542643a0c3c011ea91214340fb59264c8068847aa" Jul 17 22:19:27.695551 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:19:27.695212 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:19:40.694783 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:19:40.694750 2577 scope.go:117] "RemoveContainer" containerID="78d24cc9a9b1dd62d9c0736542643a0c3c011ea91214340fb59264c8068847aa" Jul 17 22:19:40.695254 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:19:40.694926 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:19:52.693989 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:19:52.693958 2577 scope.go:117] "RemoveContainer" containerID="78d24cc9a9b1dd62d9c0736542643a0c3c011ea91214340fb59264c8068847aa" Jul 17 22:19:52.694431 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:19:52.694131 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:20:06.694611 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:20:06.694575 2577 scope.go:117] "RemoveContainer" containerID="78d24cc9a9b1dd62d9c0736542643a0c3c011ea91214340fb59264c8068847aa" Jul 17 22:20:06.695052 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:20:06.694774 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:20:20.695009 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:20:20.694979 2577 scope.go:117] "RemoveContainer" containerID="78d24cc9a9b1dd62d9c0736542643a0c3c011ea91214340fb59264c8068847aa" Jul 17 22:20:21.355511 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:20:21.355403 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" event={"ID":"42b8b1a2-2552-46d0-985b-f7bc337a15bf","Type":"ContainerStarted","Data":"4c644c84caae3b2a36e226b3039f5963dc861570ee51ab2e23190fee1c9f8bd3"} Jul 17 22:20:21.355674 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:20:21.355629 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:20:22.368443 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:20:22.368410 2577 generic.go:356] "Generic (PLEG): container finished" podID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerID="4c644c84caae3b2a36e226b3039f5963dc861570ee51ab2e23190fee1c9f8bd3" exitCode=1 Jul 17 22:20:22.369087 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:20:22.368525 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" event={"ID":"42b8b1a2-2552-46d0-985b-f7bc337a15bf","Type":"ContainerDied","Data":"4c644c84caae3b2a36e226b3039f5963dc861570ee51ab2e23190fee1c9f8bd3"} Jul 17 22:20:22.369087 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:20:22.368629 2577 scope.go:117] "RemoveContainer" containerID="78d24cc9a9b1dd62d9c0736542643a0c3c011ea91214340fb59264c8068847aa" Jul 17 22:20:22.369087 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:20:22.368893 2577 scope.go:117] "RemoveContainer" containerID="4c644c84caae3b2a36e226b3039f5963dc861570ee51ab2e23190fee1c9f8bd3" Jul 17 22:20:22.369233 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:20:22.369141 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:20:23.376428 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:20:23.376390 2577 scope.go:117] "RemoveContainer" containerID="4c644c84caae3b2a36e226b3039f5963dc861570ee51ab2e23190fee1c9f8bd3" Jul 17 22:20:23.376833 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:20:23.376629 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:20:29.646525 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:20:29.646431 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:20:29.646945 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:20:29.646825 2577 scope.go:117] "RemoveContainer" containerID="4c644c84caae3b2a36e226b3039f5963dc861570ee51ab2e23190fee1c9f8bd3" Jul 17 22:20:29.647006 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:20:29.646987 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:20:42.694231 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:20:42.694197 2577 scope.go:117] "RemoveContainer" containerID="4c644c84caae3b2a36e226b3039f5963dc861570ee51ab2e23190fee1c9f8bd3" Jul 17 22:20:42.694665 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:20:42.694402 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:20:57.694756 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:20:57.694720 2577 scope.go:117] "RemoveContainer" containerID="4c644c84caae3b2a36e226b3039f5963dc861570ee51ab2e23190fee1c9f8bd3" Jul 17 22:20:57.695361 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:20:57.694899 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:21:08.694757 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:21:08.694720 2577 scope.go:117] "RemoveContainer" containerID="4c644c84caae3b2a36e226b3039f5963dc861570ee51ab2e23190fee1c9f8bd3" Jul 17 22:21:08.695206 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:21:08.694935 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:21:22.694046 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:21:22.694018 2577 scope.go:117] "RemoveContainer" containerID="4c644c84caae3b2a36e226b3039f5963dc861570ee51ab2e23190fee1c9f8bd3" Jul 17 22:21:22.694476 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:21:22.694200 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:21:33.698551 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:21:33.698518 2577 scope.go:117] "RemoveContainer" containerID="4c644c84caae3b2a36e226b3039f5963dc861570ee51ab2e23190fee1c9f8bd3" Jul 17 22:21:33.698965 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:21:33.698721 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:21:47.694632 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:21:47.694597 2577 scope.go:117] "RemoveContainer" containerID="4c644c84caae3b2a36e226b3039f5963dc861570ee51ab2e23190fee1c9f8bd3" Jul 17 22:21:47.695068 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:21:47.694837 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:22:02.694435 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:22:02.694355 2577 scope.go:117] "RemoveContainer" containerID="4c644c84caae3b2a36e226b3039f5963dc861570ee51ab2e23190fee1c9f8bd3" Jul 17 22:22:02.694922 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:22:02.694584 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:22:17.694284 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:22:17.694249 2577 scope.go:117] "RemoveContainer" containerID="4c644c84caae3b2a36e226b3039f5963dc861570ee51ab2e23190fee1c9f8bd3" Jul 17 22:22:17.694699 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:22:17.694476 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:22:30.694483 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:22:30.694450 2577 scope.go:117] "RemoveContainer" containerID="4c644c84caae3b2a36e226b3039f5963dc861570ee51ab2e23190fee1c9f8bd3" Jul 17 22:22:30.694975 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:22:30.694665 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:22:42.695442 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:22:42.695407 2577 scope.go:117] "RemoveContainer" containerID="4c644c84caae3b2a36e226b3039f5963dc861570ee51ab2e23190fee1c9f8bd3" Jul 17 22:22:42.695901 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:22:42.695573 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:22:55.694961 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:22:55.694926 2577 scope.go:117] "RemoveContainer" containerID="4c644c84caae3b2a36e226b3039f5963dc861570ee51ab2e23190fee1c9f8bd3" Jul 17 22:22:55.695338 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:22:55.695164 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:23:06.694795 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:06.694766 2577 scope.go:117] "RemoveContainer" containerID="4c644c84caae3b2a36e226b3039f5963dc861570ee51ab2e23190fee1c9f8bd3" Jul 17 22:23:07.223285 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:07.223246 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" event={"ID":"42b8b1a2-2552-46d0-985b-f7bc337a15bf","Type":"ContainerStarted","Data":"c4b166328ccd5a5cd602f75daffc5040bb7b99f1cb171a07c15ce6b4fcf47fee"} Jul 17 22:23:07.223482 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:07.223464 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:23:08.230094 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:08.230057 2577 generic.go:356] "Generic (PLEG): container finished" podID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerID="c4b166328ccd5a5cd602f75daffc5040bb7b99f1cb171a07c15ce6b4fcf47fee" exitCode=1 Jul 17 22:23:08.230561 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:08.230121 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" event={"ID":"42b8b1a2-2552-46d0-985b-f7bc337a15bf","Type":"ContainerDied","Data":"c4b166328ccd5a5cd602f75daffc5040bb7b99f1cb171a07c15ce6b4fcf47fee"} Jul 17 22:23:08.230561 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:08.230164 2577 scope.go:117] "RemoveContainer" containerID="4c644c84caae3b2a36e226b3039f5963dc861570ee51ab2e23190fee1c9f8bd3" Jul 17 22:23:08.230561 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:08.230358 2577 scope.go:117] "RemoveContainer" containerID="c4b166328ccd5a5cd602f75daffc5040bb7b99f1cb171a07c15ce6b4fcf47fee" Jul 17 22:23:08.230687 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:23:08.230612 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:23:09.237375 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:09.237348 2577 scope.go:117] "RemoveContainer" containerID="c4b166328ccd5a5cd602f75daffc5040bb7b99f1cb171a07c15ce6b4fcf47fee" Jul 17 22:23:09.237774 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:23:09.237550 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:23:09.647049 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:09.646940 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:23:10.242071 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:10.242044 2577 scope.go:117] "RemoveContainer" containerID="c4b166328ccd5a5cd602f75daffc5040bb7b99f1cb171a07c15ce6b4fcf47fee" Jul 17 22:23:10.242465 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:23:10.242220 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:23:21.694684 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:21.694650 2577 scope.go:117] "RemoveContainer" containerID="c4b166328ccd5a5cd602f75daffc5040bb7b99f1cb171a07c15ce6b4fcf47fee" Jul 17 22:23:21.695171 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:23:21.694839 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=maas-api pod=maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc_odh-ai-gateway-infra(42b8b1a2-2552-46d0-985b-f7bc337a15bf)\"" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" Jul 17 22:23:29.993485 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:29.993396 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc"] Jul 17 22:23:30.140789 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:30.140763 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:23:30.211060 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:30.211030 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8ljr\" (UniqueName: \"kubernetes.io/projected/42b8b1a2-2552-46d0-985b-f7bc337a15bf-kube-api-access-b8ljr\") pod \"42b8b1a2-2552-46d0-985b-f7bc337a15bf\" (UID: \"42b8b1a2-2552-46d0-985b-f7bc337a15bf\") " Jul 17 22:23:30.211199 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:30.211104 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42b8b1a2-2552-46d0-985b-f7bc337a15bf-maas-api-tls\") pod \"42b8b1a2-2552-46d0-985b-f7bc337a15bf\" (UID: \"42b8b1a2-2552-46d0-985b-f7bc337a15bf\") " Jul 17 22:23:30.211199 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:30.211148 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b8b1a2-2552-46d0-985b-f7bc337a15bf-openshift-service-ca\") pod \"42b8b1a2-2552-46d0-985b-f7bc337a15bf\" (UID: \"42b8b1a2-2552-46d0-985b-f7bc337a15bf\") " Jul 17 22:23:30.211551 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:30.211526 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b8b1a2-2552-46d0-985b-f7bc337a15bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "42b8b1a2-2552-46d0-985b-f7bc337a15bf" (UID: "42b8b1a2-2552-46d0-985b-f7bc337a15bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:23:30.213261 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:30.213235 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b8b1a2-2552-46d0-985b-f7bc337a15bf-kube-api-access-b8ljr" (OuterVolumeSpecName: "kube-api-access-b8ljr") pod "42b8b1a2-2552-46d0-985b-f7bc337a15bf" (UID: "42b8b1a2-2552-46d0-985b-f7bc337a15bf"). InnerVolumeSpecName "kube-api-access-b8ljr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:23:30.213368 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:30.213311 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42b8b1a2-2552-46d0-985b-f7bc337a15bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "42b8b1a2-2552-46d0-985b-f7bc337a15bf" (UID: "42b8b1a2-2552-46d0-985b-f7bc337a15bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:23:30.312890 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:30.312825 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42b8b1a2-2552-46d0-985b-f7bc337a15bf-maas-api-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:23:30.312890 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:30.312850 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42b8b1a2-2552-46d0-985b-f7bc337a15bf-openshift-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:23:30.312890 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:30.312861 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8ljr\" (UniqueName: \"kubernetes.io/projected/42b8b1a2-2552-46d0-985b-f7bc337a15bf-kube-api-access-b8ljr\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:23:30.345973 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:30.345939 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" event={"ID":"42b8b1a2-2552-46d0-985b-f7bc337a15bf","Type":"ContainerDied","Data":"0e49ecef1c51be8cdba7d92a432cccc8f68533a000846191bf2f7894d986ab96"} Jul 17 22:23:30.345973 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:30.345969 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc" Jul 17 22:23:30.346155 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:30.346034 2577 scope.go:117] "RemoveContainer" containerID="c4b166328ccd5a5cd602f75daffc5040bb7b99f1cb171a07c15ce6b4fcf47fee" Jul 17 22:23:30.367796 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:30.367772 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc"] Jul 17 22:23:30.374012 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:30.373991 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-45495ac6-869d7cd756-8nwnc"] Jul 17 22:23:31.700176 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:31.700134 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" path="/var/lib/kubelet/pods/42b8b1a2-2552-46d0-985b-f7bc337a15bf/volumes" Jul 17 22:23:38.057852 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.057822 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg"] Jul 17 22:23:38.058355 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.058225 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:23:38.058355 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.058235 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:23:38.058355 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.058252 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:23:38.058355 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.058257 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:23:38.058355 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.058264 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:23:38.058355 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.058270 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:23:38.058355 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.058277 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:23:38.058355 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.058282 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:23:38.058355 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.058293 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:23:38.058355 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.058298 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:23:38.058355 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.058304 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" containerName="maas-api" Jul 17 22:23:38.058355 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.058309 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" containerName="maas-api" Jul 17 22:23:38.058813 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.058381 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:23:38.058813 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.058387 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:23:38.058813 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.058392 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:23:38.058813 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.058400 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c2abcd7-1ef6-4826-ae84-5cf9bb19fb70" containerName="maas-api" Jul 17 22:23:38.058813 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.058406 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:23:38.061366 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.061350 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:23:38.065606 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.065586 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ded85f30-serving-cert\"" Jul 17 22:23:38.086719 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.086691 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg"] Jul 17 22:23:38.188013 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.187969 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/815e3f00-fe5f-44ae-a4d9-05facabccfa1-maas-api-tls\") pod \"maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg\" (UID: \"815e3f00-fe5f-44ae-a4d9-05facabccfa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:23:38.188182 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.188018 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcvmp\" (UniqueName: \"kubernetes.io/projected/815e3f00-fe5f-44ae-a4d9-05facabccfa1-kube-api-access-bcvmp\") pod \"maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg\" (UID: \"815e3f00-fe5f-44ae-a4d9-05facabccfa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:23:38.188182 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.188081 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/815e3f00-fe5f-44ae-a4d9-05facabccfa1-openshift-service-ca\") pod \"maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg\" (UID: \"815e3f00-fe5f-44ae-a4d9-05facabccfa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:23:38.288939 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.288896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/815e3f00-fe5f-44ae-a4d9-05facabccfa1-maas-api-tls\") pod \"maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg\" (UID: \"815e3f00-fe5f-44ae-a4d9-05facabccfa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:23:38.289144 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.288952 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcvmp\" (UniqueName: \"kubernetes.io/projected/815e3f00-fe5f-44ae-a4d9-05facabccfa1-kube-api-access-bcvmp\") pod \"maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg\" (UID: \"815e3f00-fe5f-44ae-a4d9-05facabccfa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:23:38.289144 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.288998 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/815e3f00-fe5f-44ae-a4d9-05facabccfa1-openshift-service-ca\") pod \"maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg\" (UID: \"815e3f00-fe5f-44ae-a4d9-05facabccfa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:23:38.289700 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.289679 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/815e3f00-fe5f-44ae-a4d9-05facabccfa1-openshift-service-ca\") pod \"maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg\" (UID: \"815e3f00-fe5f-44ae-a4d9-05facabccfa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:23:38.291472 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.291455 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/815e3f00-fe5f-44ae-a4d9-05facabccfa1-maas-api-tls\") pod \"maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg\" (UID: \"815e3f00-fe5f-44ae-a4d9-05facabccfa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:23:38.300518 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.300466 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcvmp\" (UniqueName: \"kubernetes.io/projected/815e3f00-fe5f-44ae-a4d9-05facabccfa1-kube-api-access-bcvmp\") pod \"maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg\" (UID: \"815e3f00-fe5f-44ae-a4d9-05facabccfa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:23:38.371422 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.371357 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:23:38.519352 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.519328 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg"] Jul 17 22:23:38.521446 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:23:38.521425 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815e3f00_fe5f_44ae_a4d9_05facabccfa1.slice/crio-3e796eb8b69b60d677b76dbde8ebb0156ee57fe200d0b1a01798a5362fc05f0e WatchSource:0}: Error finding container 3e796eb8b69b60d677b76dbde8ebb0156ee57fe200d0b1a01798a5362fc05f0e: Status 404 returned error can't find the container with id 3e796eb8b69b60d677b76dbde8ebb0156ee57fe200d0b1a01798a5362fc05f0e Jul 17 22:23:38.522770 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:38.522751 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 22:23:39.402800 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:39.402751 2577 generic.go:356] "Generic (PLEG): container finished" podID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerID="48e763cf47f3ca325121d8687e47363a75bb1e8cb2e113ffcb66b8c22cd0f498" exitCode=1 Jul 17 22:23:39.403268 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:39.402845 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" event={"ID":"815e3f00-fe5f-44ae-a4d9-05facabccfa1","Type":"ContainerDied","Data":"48e763cf47f3ca325121d8687e47363a75bb1e8cb2e113ffcb66b8c22cd0f498"} Jul 17 22:23:39.403268 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:39.402930 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" event={"ID":"815e3f00-fe5f-44ae-a4d9-05facabccfa1","Type":"ContainerStarted","Data":"3e796eb8b69b60d677b76dbde8ebb0156ee57fe200d0b1a01798a5362fc05f0e"} Jul 17 22:23:39.403268 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:39.403015 2577 scope.go:117] "RemoveContainer" containerID="48e763cf47f3ca325121d8687e47363a75bb1e8cb2e113ffcb66b8c22cd0f498" Jul 17 22:23:40.409651 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:40.409572 2577 generic.go:356] "Generic (PLEG): container finished" podID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerID="487abce655466247158d5eceabf7a915f94f843c6360fe1d8531b404a5465b1a" exitCode=1 Jul 17 22:23:40.409651 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:40.409603 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" event={"ID":"815e3f00-fe5f-44ae-a4d9-05facabccfa1","Type":"ContainerDied","Data":"487abce655466247158d5eceabf7a915f94f843c6360fe1d8531b404a5465b1a"} Jul 17 22:23:40.410122 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:40.409666 2577 scope.go:117] "RemoveContainer" containerID="48e763cf47f3ca325121d8687e47363a75bb1e8cb2e113ffcb66b8c22cd0f498" Jul 17 22:23:40.410122 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:40.409882 2577 scope.go:117] "RemoveContainer" containerID="487abce655466247158d5eceabf7a915f94f843c6360fe1d8531b404a5465b1a" Jul 17 22:23:40.410122 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:23:40.410109 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:23:41.416539 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:41.416485 2577 scope.go:117] "RemoveContainer" containerID="487abce655466247158d5eceabf7a915f94f843c6360fe1d8531b404a5465b1a" Jul 17 22:23:41.416932 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:23:41.416689 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:23:43.371882 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:43.371856 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:23:43.372263 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:43.372196 2577 scope.go:117] "RemoveContainer" containerID="487abce655466247158d5eceabf7a915f94f843c6360fe1d8531b404a5465b1a" Jul 17 22:23:43.372375 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:23:43.372356 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:23:48.371656 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:48.371622 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:23:48.372046 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:48.372009 2577 scope.go:117] "RemoveContainer" containerID="487abce655466247158d5eceabf7a915f94f843c6360fe1d8531b404a5465b1a" Jul 17 22:23:48.372200 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:23:48.372180 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 10s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:23:58.695040 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:58.695011 2577 scope.go:117] "RemoveContainer" containerID="487abce655466247158d5eceabf7a915f94f843c6360fe1d8531b404a5465b1a" Jul 17 22:23:59.511161 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:59.511132 2577 generic.go:356] "Generic (PLEG): container finished" podID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerID="6706e6ab8e95aa5fd03b8fb17040c8ec22939dd370d7dc98e6822739bd3e67d6" exitCode=1 Jul 17 22:23:59.511305 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:59.511200 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" event={"ID":"815e3f00-fe5f-44ae-a4d9-05facabccfa1","Type":"ContainerDied","Data":"6706e6ab8e95aa5fd03b8fb17040c8ec22939dd370d7dc98e6822739bd3e67d6"} Jul 17 22:23:59.511305 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:59.511260 2577 scope.go:117] "RemoveContainer" containerID="487abce655466247158d5eceabf7a915f94f843c6360fe1d8531b404a5465b1a" Jul 17 22:23:59.511682 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:23:59.511661 2577 scope.go:117] "RemoveContainer" containerID="6706e6ab8e95aa5fd03b8fb17040c8ec22939dd370d7dc98e6822739bd3e67d6" Jul 17 22:23:59.511896 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:23:59.511878 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:24:03.372423 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:24:03.372376 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:24:03.372881 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:24:03.372858 2577 scope.go:117] "RemoveContainer" containerID="6706e6ab8e95aa5fd03b8fb17040c8ec22939dd370d7dc98e6822739bd3e67d6" Jul 17 22:24:03.373101 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:24:03.373081 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:24:08.371698 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:24:08.371657 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:24:08.372062 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:24:08.372052 2577 scope.go:117] "RemoveContainer" containerID="6706e6ab8e95aa5fd03b8fb17040c8ec22939dd370d7dc98e6822739bd3e67d6" Jul 17 22:24:08.372245 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:24:08.372224 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 20s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:24:22.694774 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:24:22.694744 2577 scope.go:117] "RemoveContainer" containerID="6706e6ab8e95aa5fd03b8fb17040c8ec22939dd370d7dc98e6822739bd3e67d6" Jul 17 22:24:23.648064 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:24:23.648031 2577 generic.go:356] "Generic (PLEG): container finished" podID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerID="40311c086d815371fa2791186cf991fc5c725a69a62af887e04d9d282cd4a08d" exitCode=1 Jul 17 22:24:23.648243 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:24:23.648102 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" event={"ID":"815e3f00-fe5f-44ae-a4d9-05facabccfa1","Type":"ContainerDied","Data":"40311c086d815371fa2791186cf991fc5c725a69a62af887e04d9d282cd4a08d"} Jul 17 22:24:23.648243 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:24:23.648162 2577 scope.go:117] "RemoveContainer" containerID="6706e6ab8e95aa5fd03b8fb17040c8ec22939dd370d7dc98e6822739bd3e67d6" Jul 17 22:24:23.648585 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:24:23.648565 2577 scope.go:117] "RemoveContainer" containerID="40311c086d815371fa2791186cf991fc5c725a69a62af887e04d9d282cd4a08d" Jul 17 22:24:23.648793 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:24:23.648772 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:24:28.371642 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:24:28.371599 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:24:28.371642 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:24:28.371660 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:24:28.372077 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:24:28.371969 2577 scope.go:117] "RemoveContainer" containerID="40311c086d815371fa2791186cf991fc5c725a69a62af887e04d9d282cd4a08d" Jul 17 22:24:28.372149 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:24:28.372128 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:24:43.697594 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:24:43.697558 2577 scope.go:117] "RemoveContainer" containerID="40311c086d815371fa2791186cf991fc5c725a69a62af887e04d9d282cd4a08d" Jul 17 22:24:43.698001 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:24:43.697785 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:24:56.694628 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:24:56.694545 2577 scope.go:117] "RemoveContainer" containerID="40311c086d815371fa2791186cf991fc5c725a69a62af887e04d9d282cd4a08d" Jul 17 22:24:56.695097 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:24:56.694722 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 40s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:25:10.694608 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:25:10.694576 2577 scope.go:117] "RemoveContainer" containerID="40311c086d815371fa2791186cf991fc5c725a69a62af887e04d9d282cd4a08d" Jul 17 22:25:11.900594 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:25:11.900561 2577 generic.go:356] "Generic (PLEG): container finished" podID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerID="48bd28363baa27a1a4703b5a1c51400f6e7e330aa1599925f77325ba12749523" exitCode=1 Jul 17 22:25:11.901016 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:25:11.900613 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" event={"ID":"815e3f00-fe5f-44ae-a4d9-05facabccfa1","Type":"ContainerDied","Data":"48bd28363baa27a1a4703b5a1c51400f6e7e330aa1599925f77325ba12749523"} Jul 17 22:25:11.901016 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:25:11.900667 2577 scope.go:117] "RemoveContainer" containerID="40311c086d815371fa2791186cf991fc5c725a69a62af887e04d9d282cd4a08d" Jul 17 22:25:11.901016 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:25:11.900967 2577 scope.go:117] "RemoveContainer" containerID="48bd28363baa27a1a4703b5a1c51400f6e7e330aa1599925f77325ba12749523" Jul 17 22:25:11.901197 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:25:11.901172 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:25:13.372322 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:25:13.372284 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:25:13.372777 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:25:13.372694 2577 scope.go:117] "RemoveContainer" containerID="48bd28363baa27a1a4703b5a1c51400f6e7e330aa1599925f77325ba12749523" Jul 17 22:25:13.372878 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:25:13.372859 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:25:18.372345 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:25:18.372306 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:25:18.372792 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:25:18.372686 2577 scope.go:117] "RemoveContainer" containerID="48bd28363baa27a1a4703b5a1c51400f6e7e330aa1599925f77325ba12749523" Jul 17 22:25:18.372888 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:25:18.372870 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:25:33.697457 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:25:33.697430 2577 scope.go:117] "RemoveContainer" containerID="48bd28363baa27a1a4703b5a1c51400f6e7e330aa1599925f77325ba12749523" Jul 17 22:25:33.697962 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:25:33.697653 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:25:46.694903 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:25:46.694873 2577 scope.go:117] "RemoveContainer" containerID="48bd28363baa27a1a4703b5a1c51400f6e7e330aa1599925f77325ba12749523" Jul 17 22:25:46.695295 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:25:46.695050 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:26:01.694789 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:01.694756 2577 scope.go:117] "RemoveContainer" containerID="48bd28363baa27a1a4703b5a1c51400f6e7e330aa1599925f77325ba12749523" Jul 17 22:26:01.695474 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:26:01.695426 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:26:13.697334 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:13.697308 2577 scope.go:117] "RemoveContainer" containerID="48bd28363baa27a1a4703b5a1c51400f6e7e330aa1599925f77325ba12749523" Jul 17 22:26:13.697875 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:26:13.697523 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:26:25.694936 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:25.694863 2577 scope.go:117] "RemoveContainer" containerID="48bd28363baa27a1a4703b5a1c51400f6e7e330aa1599925f77325ba12749523" Jul 17 22:26:25.695292 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:26:25.695052 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 1m20s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:26:39.694723 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:39.694689 2577 scope.go:117] "RemoveContainer" containerID="48bd28363baa27a1a4703b5a1c51400f6e7e330aa1599925f77325ba12749523" Jul 17 22:26:40.351826 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:40.351736 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" event={"ID":"815e3f00-fe5f-44ae-a4d9-05facabccfa1","Type":"ContainerStarted","Data":"ef851692e0d5b7b570c9c514784de5e38d142c2fee2e2cda29842aa25aa9d77d"} Jul 17 22:26:40.352221 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:40.351959 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:26:40.369011 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:40.368967 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podStartSLOduration=180.172070029 podStartE2EDuration="3m2.368953057s" totalImagesPullingTime="2.196883028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:23:38 +0000 UTC" imagePullSessionsCount=6 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:26:40.367812909 +0000 UTC m=+2437.228036410" watchObservedRunningTime="2026-07-17 22:26:40.368953057 +0000 UTC m=+2437.229176558" Jul 17 22:26:41.358594 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:41.358562 2577 generic.go:356] "Generic (PLEG): container finished" podID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerID="ef851692e0d5b7b570c9c514784de5e38d142c2fee2e2cda29842aa25aa9d77d" exitCode=1 Jul 17 22:26:41.359027 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:41.358666 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" event={"ID":"815e3f00-fe5f-44ae-a4d9-05facabccfa1","Type":"ContainerDied","Data":"ef851692e0d5b7b570c9c514784de5e38d142c2fee2e2cda29842aa25aa9d77d"} Jul 17 22:26:41.359027 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:41.358724 2577 scope.go:117] "RemoveContainer" containerID="48bd28363baa27a1a4703b5a1c51400f6e7e330aa1599925f77325ba12749523" Jul 17 22:26:41.359027 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:41.358878 2577 scope.go:117] "RemoveContainer" containerID="ef851692e0d5b7b570c9c514784de5e38d142c2fee2e2cda29842aa25aa9d77d" Jul 17 22:26:41.359169 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:26:41.359112 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:26:42.365631 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:42.365604 2577 scope.go:117] "RemoveContainer" containerID="ef851692e0d5b7b570c9c514784de5e38d142c2fee2e2cda29842aa25aa9d77d" Jul 17 22:26:42.366011 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:26:42.365830 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:26:48.371745 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:48.371712 2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:26:48.372161 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:48.372075 2577 scope.go:117] "RemoveContainer" containerID="ef851692e0d5b7b570c9c514784de5e38d142c2fee2e2cda29842aa25aa9d77d" Jul 17 22:26:48.372259 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:26:48.372240 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"maas-api\" with CrashLoopBackOff: \"back-off 2m40s restarting failed container=maas-api pod=maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg_odh-ai-gateway-infra(815e3f00-fe5f-44ae-a4d9-05facabccfa1)\"" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" Jul 17 22:26:48.468569 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:26:48.468534 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-serving-cert: secret "maas-api-e2e-derive-ded85f30-serving-cert" not found Jul 17 22:26:48.468762 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:26:48.468625 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/815e3f00-fe5f-44ae-a4d9-05facabccfa1-maas-api-tls podName:815e3f00-fe5f-44ae-a4d9-05facabccfa1 nodeName:}" failed. No retries permitted until 2026-07-17 22:26:48.968601559 +0000 UTC m=+2445.828825062 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/815e3f00-fe5f-44ae-a4d9-05facabccfa1-maas-api-tls") pod "maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" (UID: "815e3f00-fe5f-44ae-a4d9-05facabccfa1") : secret "maas-api-e2e-derive-ded85f30-serving-cert" not found Jul 17 22:26:48.474091 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:48.474060 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg"] Jul 17 22:26:48.617664 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:48.617637 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:26:48.669785 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:48.669707 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/815e3f00-fe5f-44ae-a4d9-05facabccfa1-maas-api-tls\") pod \"815e3f00-fe5f-44ae-a4d9-05facabccfa1\" (UID: \"815e3f00-fe5f-44ae-a4d9-05facabccfa1\") " Jul 17 22:26:48.669785 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:48.669746 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcvmp\" (UniqueName: \"kubernetes.io/projected/815e3f00-fe5f-44ae-a4d9-05facabccfa1-kube-api-access-bcvmp\") pod \"815e3f00-fe5f-44ae-a4d9-05facabccfa1\" (UID: \"815e3f00-fe5f-44ae-a4d9-05facabccfa1\") " Jul 17 22:26:48.670018 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:48.669879 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/815e3f00-fe5f-44ae-a4d9-05facabccfa1-openshift-service-ca\") pod \"815e3f00-fe5f-44ae-a4d9-05facabccfa1\" (UID: \"815e3f00-fe5f-44ae-a4d9-05facabccfa1\") " Jul 17 22:26:48.670207 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:48.670185 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/815e3f00-fe5f-44ae-a4d9-05facabccfa1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "815e3f00-fe5f-44ae-a4d9-05facabccfa1" (UID: "815e3f00-fe5f-44ae-a4d9-05facabccfa1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:26:48.671928 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:48.671905 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/815e3f00-fe5f-44ae-a4d9-05facabccfa1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "815e3f00-fe5f-44ae-a4d9-05facabccfa1" (UID: "815e3f00-fe5f-44ae-a4d9-05facabccfa1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:26:48.672046 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:48.671980 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815e3f00-fe5f-44ae-a4d9-05facabccfa1-kube-api-access-bcvmp" (OuterVolumeSpecName: "kube-api-access-bcvmp") pod "815e3f00-fe5f-44ae-a4d9-05facabccfa1" (UID: "815e3f00-fe5f-44ae-a4d9-05facabccfa1"). InnerVolumeSpecName "kube-api-access-bcvmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:26:48.771555 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:48.771485 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/815e3f00-fe5f-44ae-a4d9-05facabccfa1-maas-api-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:26:48.771555 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:48.771555 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcvmp\" (UniqueName: \"kubernetes.io/projected/815e3f00-fe5f-44ae-a4d9-05facabccfa1-kube-api-access-bcvmp\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:26:48.771777 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:48.771570 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/815e3f00-fe5f-44ae-a4d9-05facabccfa1-openshift-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:26:49.403533 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:49.403508 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" Jul 17 22:26:49.403917 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:49.403502 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg" event={"ID":"815e3f00-fe5f-44ae-a4d9-05facabccfa1","Type":"ContainerDied","Data":"3e796eb8b69b60d677b76dbde8ebb0156ee57fe200d0b1a01798a5362fc05f0e"} Jul 17 22:26:49.403917 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:49.403647 2577 scope.go:117] "RemoveContainer" containerID="ef851692e0d5b7b570c9c514784de5e38d142c2fee2e2cda29842aa25aa9d77d" Jul 17 22:26:49.425042 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:49.425018 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg"] Jul 17 22:26:49.428942 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:49.428919 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ded85f30-79b7c97c99-xw7rg"] Jul 17 22:26:49.701172 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:26:49.701132 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" path="/var/lib/kubelet/pods/815e3f00-fe5f-44ae-a4d9-05facabccfa1/volumes" Jul 17 22:28:33.880796 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:33.880762 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr"] Jul 17 22:28:33.881285 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:33.881242 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerName="maas-api" Jul 17 22:28:33.881285 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:33.881258 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerName="maas-api" Jul 17 22:28:33.881285 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:33.881269 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerName="maas-api" Jul 17 22:28:33.881285 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:33.881275 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerName="maas-api" Jul 17 22:28:33.881285 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:33.881283 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerName="maas-api" Jul 17 22:28:33.881285 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:33.881289 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerName="maas-api" Jul 17 22:28:33.881532 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:33.881296 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:28:33.881532 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:33.881301 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:28:33.881532 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:33.881308 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerName="maas-api" Jul 17 22:28:33.881532 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:33.881314 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerName="maas-api" Jul 17 22:28:33.881532 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:33.881386 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:28:33.881532 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:33.881392 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerName="maas-api" Jul 17 22:28:33.881532 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:33.881397 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerName="maas-api" Jul 17 22:28:33.881532 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:33.881403 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerName="maas-api" Jul 17 22:28:33.881532 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:33.881411 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:28:33.884183 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:33.884161 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" Jul 17 22:28:33.887719 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:33.887696 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c8731528-serving-cert\"" Jul 17 22:28:33.895948 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:33.895927 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr"] Jul 17 22:28:34.061586 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:34.061541 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16697a5c-4485-4b70-9e12-34f074194db7-maas-api-tls\") pod \"maas-api-e2e-mt-c8731528-7fd9894787-j2cqr\" (UID: \"16697a5c-4485-4b70-9e12-34f074194db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" Jul 17 22:28:34.061777 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:34.061596 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16697a5c-4485-4b70-9e12-34f074194db7-openshift-service-ca\") pod \"maas-api-e2e-mt-c8731528-7fd9894787-j2cqr\" (UID: \"16697a5c-4485-4b70-9e12-34f074194db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" Jul 17 22:28:34.061777 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:34.061661 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkbzv\" (UniqueName: \"kubernetes.io/projected/16697a5c-4485-4b70-9e12-34f074194db7-kube-api-access-jkbzv\") pod \"maas-api-e2e-mt-c8731528-7fd9894787-j2cqr\" (UID: \"16697a5c-4485-4b70-9e12-34f074194db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" Jul 17 22:28:34.162994 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:34.162894 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16697a5c-4485-4b70-9e12-34f074194db7-maas-api-tls\") pod \"maas-api-e2e-mt-c8731528-7fd9894787-j2cqr\" (UID: \"16697a5c-4485-4b70-9e12-34f074194db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" Jul 17 22:28:34.162994 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:34.162936 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16697a5c-4485-4b70-9e12-34f074194db7-openshift-service-ca\") pod \"maas-api-e2e-mt-c8731528-7fd9894787-j2cqr\" (UID: \"16697a5c-4485-4b70-9e12-34f074194db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" Jul 17 22:28:34.162994 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:34.162974 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkbzv\" (UniqueName: \"kubernetes.io/projected/16697a5c-4485-4b70-9e12-34f074194db7-kube-api-access-jkbzv\") pod \"maas-api-e2e-mt-c8731528-7fd9894787-j2cqr\" (UID: \"16697a5c-4485-4b70-9e12-34f074194db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" Jul 17 22:28:34.163287 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:28:34.163047 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-serving-cert: secret "maas-api-e2e-mt-c8731528-serving-cert" not found Jul 17 22:28:34.163287 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:28:34.163118 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16697a5c-4485-4b70-9e12-34f074194db7-maas-api-tls podName:16697a5c-4485-4b70-9e12-34f074194db7 nodeName:}" failed. No retries permitted until 2026-07-17 22:28:34.663101307 +0000 UTC m=+2551.523324788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16697a5c-4485-4b70-9e12-34f074194db7-maas-api-tls") pod "maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" (UID: "16697a5c-4485-4b70-9e12-34f074194db7") : secret "maas-api-e2e-mt-c8731528-serving-cert" not found Jul 17 22:28:34.163587 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:34.163569 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16697a5c-4485-4b70-9e12-34f074194db7-openshift-service-ca\") pod \"maas-api-e2e-mt-c8731528-7fd9894787-j2cqr\" (UID: \"16697a5c-4485-4b70-9e12-34f074194db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" Jul 17 22:28:34.172037 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:34.172012 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkbzv\" (UniqueName: \"kubernetes.io/projected/16697a5c-4485-4b70-9e12-34f074194db7-kube-api-access-jkbzv\") pod \"maas-api-e2e-mt-c8731528-7fd9894787-j2cqr\" (UID: \"16697a5c-4485-4b70-9e12-34f074194db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" Jul 17 22:28:34.668156 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:34.668122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16697a5c-4485-4b70-9e12-34f074194db7-maas-api-tls\") pod \"maas-api-e2e-mt-c8731528-7fd9894787-j2cqr\" (UID: \"16697a5c-4485-4b70-9e12-34f074194db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" Jul 17 22:28:34.670754 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:34.670728 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16697a5c-4485-4b70-9e12-34f074194db7-maas-api-tls\") pod \"maas-api-e2e-mt-c8731528-7fd9894787-j2cqr\" (UID: \"16697a5c-4485-4b70-9e12-34f074194db7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" Jul 17 22:28:34.797211 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:34.797181 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" Jul 17 22:28:35.154671 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:35.154644 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr"] Jul 17 22:28:35.156741 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:28:35.156703 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16697a5c_4485_4b70_9e12_34f074194db7.slice/crio-c1d08ef366cce1293cd4a06021da13a36fd24752597baef29604474f6b57be80 WatchSource:0}: Error finding container c1d08ef366cce1293cd4a06021da13a36fd24752597baef29604474f6b57be80: Status 404 returned error can't find the container with id c1d08ef366cce1293cd4a06021da13a36fd24752597baef29604474f6b57be80 Jul 17 22:28:35.956364 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:35.956335 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" event={"ID":"16697a5c-4485-4b70-9e12-34f074194db7","Type":"ContainerStarted","Data":"2361d2147f5ab0b1dcc2b949c4d941cfe3f01484c89238c350678670874c1179"} Jul 17 22:28:35.956731 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:35.956383 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" event={"ID":"16697a5c-4485-4b70-9e12-34f074194db7","Type":"ContainerStarted","Data":"c1d08ef366cce1293cd4a06021da13a36fd24752597baef29604474f6b57be80"} Jul 17 22:28:35.956731 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:35.956422 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" Jul 17 22:28:35.974721 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:35.974670 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" podStartSLOduration=2.603082139 podStartE2EDuration="2.974653414s" totalImagesPullingTime="371.571275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:28:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:28:35.973881723 +0000 UTC m=+2552.834105246" watchObservedRunningTime="2026-07-17 22:28:35.974653414 +0000 UTC m=+2552.834876954" Jul 17 22:28:41.967320 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:41.967291 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" Jul 17 22:28:56.013607 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:56.013575 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr"] Jul 17 22:28:56.014111 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:56.013795 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" podUID="16697a5c-4485-4b70-9e12-34f074194db7" containerName="maas-api" containerID="cri-o://2361d2147f5ab0b1dcc2b949c4d941cfe3f01484c89238c350678670874c1179" gracePeriod=30 Jul 17 22:28:56.261767 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:56.261738 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" Jul 17 22:28:56.367202 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:56.367121 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16697a5c-4485-4b70-9e12-34f074194db7-maas-api-tls\") pod \"16697a5c-4485-4b70-9e12-34f074194db7\" (UID: \"16697a5c-4485-4b70-9e12-34f074194db7\") " Jul 17 22:28:56.367343 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:56.367227 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkbzv\" (UniqueName: \"kubernetes.io/projected/16697a5c-4485-4b70-9e12-34f074194db7-kube-api-access-jkbzv\") pod \"16697a5c-4485-4b70-9e12-34f074194db7\" (UID: \"16697a5c-4485-4b70-9e12-34f074194db7\") " Jul 17 22:28:56.367343 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:56.367297 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16697a5c-4485-4b70-9e12-34f074194db7-openshift-service-ca\") pod \"16697a5c-4485-4b70-9e12-34f074194db7\" (UID: \"16697a5c-4485-4b70-9e12-34f074194db7\") " Jul 17 22:28:56.367642 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:56.367612 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16697a5c-4485-4b70-9e12-34f074194db7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "16697a5c-4485-4b70-9e12-34f074194db7" (UID: "16697a5c-4485-4b70-9e12-34f074194db7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:28:56.369266 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:56.369247 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16697a5c-4485-4b70-9e12-34f074194db7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "16697a5c-4485-4b70-9e12-34f074194db7" (UID: "16697a5c-4485-4b70-9e12-34f074194db7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:28:56.369396 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:56.369375 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16697a5c-4485-4b70-9e12-34f074194db7-kube-api-access-jkbzv" (OuterVolumeSpecName: "kube-api-access-jkbzv") pod "16697a5c-4485-4b70-9e12-34f074194db7" (UID: "16697a5c-4485-4b70-9e12-34f074194db7"). InnerVolumeSpecName "kube-api-access-jkbzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:28:56.468552 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:56.468514 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkbzv\" (UniqueName: \"kubernetes.io/projected/16697a5c-4485-4b70-9e12-34f074194db7-kube-api-access-jkbzv\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:28:56.468552 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:56.468548 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16697a5c-4485-4b70-9e12-34f074194db7-openshift-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:28:56.468552 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:56.468561 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16697a5c-4485-4b70-9e12-34f074194db7-maas-api-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:28:57.068151 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:57.068113 2577 generic.go:356] "Generic (PLEG): container finished" podID="16697a5c-4485-4b70-9e12-34f074194db7" containerID="2361d2147f5ab0b1dcc2b949c4d941cfe3f01484c89238c350678670874c1179" exitCode=0 Jul 17 22:28:57.068643 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:57.068179 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" event={"ID":"16697a5c-4485-4b70-9e12-34f074194db7","Type":"ContainerDied","Data":"2361d2147f5ab0b1dcc2b949c4d941cfe3f01484c89238c350678670874c1179"} Jul 17 22:28:57.068643 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:57.068182 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" Jul 17 22:28:57.068643 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:57.068222 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr" event={"ID":"16697a5c-4485-4b70-9e12-34f074194db7","Type":"ContainerDied","Data":"c1d08ef366cce1293cd4a06021da13a36fd24752597baef29604474f6b57be80"} Jul 17 22:28:57.068643 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:57.068243 2577 scope.go:117] "RemoveContainer" containerID="2361d2147f5ab0b1dcc2b949c4d941cfe3f01484c89238c350678670874c1179" Jul 17 22:28:57.077702 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:57.077686 2577 scope.go:117] "RemoveContainer" containerID="2361d2147f5ab0b1dcc2b949c4d941cfe3f01484c89238c350678670874c1179" Jul 17 22:28:57.077940 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:28:57.077921 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2361d2147f5ab0b1dcc2b949c4d941cfe3f01484c89238c350678670874c1179\": container with ID starting with 2361d2147f5ab0b1dcc2b949c4d941cfe3f01484c89238c350678670874c1179 not found: ID does not exist" containerID="2361d2147f5ab0b1dcc2b949c4d941cfe3f01484c89238c350678670874c1179" Jul 17 22:28:57.078005 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:57.077948 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2361d2147f5ab0b1dcc2b949c4d941cfe3f01484c89238c350678670874c1179"} err="failed to get container status \"2361d2147f5ab0b1dcc2b949c4d941cfe3f01484c89238c350678670874c1179\": rpc error: code = NotFound desc = could not find container \"2361d2147f5ab0b1dcc2b949c4d941cfe3f01484c89238c350678670874c1179\": container with ID starting with 2361d2147f5ab0b1dcc2b949c4d941cfe3f01484c89238c350678670874c1179 not found: ID does not exist" Jul 17 22:28:57.088085 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:57.088063 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr"] Jul 17 22:28:57.092447 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:57.092415 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c8731528-7fd9894787-j2cqr"] Jul 17 22:28:57.700422 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:28:57.700386 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16697a5c-4485-4b70-9e12-34f074194db7" path="/var/lib/kubelet/pods/16697a5c-4485-4b70-9e12-34f074194db7/volumes" Jul 17 22:29:34.266006 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.265920 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr"] Jul 17 22:29:34.266540 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.266410 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerName="maas-api" Jul 17 22:29:34.266540 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.266424 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerName="maas-api" Jul 17 22:29:34.266540 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.266446 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16697a5c-4485-4b70-9e12-34f074194db7" containerName="maas-api" Jul 17 22:29:34.266540 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.266453 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="16697a5c-4485-4b70-9e12-34f074194db7" containerName="maas-api" Jul 17 22:29:34.266540 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.266472 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:29:34.266540 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.266478 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:29:34.266859 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.266571 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="42b8b1a2-2552-46d0-985b-f7bc337a15bf" containerName="maas-api" Jul 17 22:29:34.266859 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.266581 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerName="maas-api" Jul 17 22:29:34.266859 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.266586 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerName="maas-api" Jul 17 22:29:34.266859 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.266592 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerName="maas-api" Jul 17 22:29:34.266859 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.266598 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="16697a5c-4485-4b70-9e12-34f074194db7" containerName="maas-api" Jul 17 22:29:34.269742 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.269723 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" Jul 17 22:29:34.271953 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.271932 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-895e50b8-serving-cert\"" Jul 17 22:29:34.284942 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.284915 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr"] Jul 17 22:29:34.332812 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.332784 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1112492-cde3-44c0-9eb0-d02d41d65997-openshift-service-ca\") pod \"maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr\" (UID: \"e1112492-cde3-44c0-9eb0-d02d41d65997\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" Jul 17 22:29:34.332988 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.332947 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1112492-cde3-44c0-9eb0-d02d41d65997-maas-api-tls\") pod \"maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr\" (UID: \"e1112492-cde3-44c0-9eb0-d02d41d65997\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" Jul 17 22:29:34.333053 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.333016 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqs57\" (UniqueName: \"kubernetes.io/projected/e1112492-cde3-44c0-9eb0-d02d41d65997-kube-api-access-bqs57\") pod \"maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr\" (UID: \"e1112492-cde3-44c0-9eb0-d02d41d65997\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" Jul 17 22:29:34.434253 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.434216 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqs57\" (UniqueName: \"kubernetes.io/projected/e1112492-cde3-44c0-9eb0-d02d41d65997-kube-api-access-bqs57\") pod \"maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr\" (UID: \"e1112492-cde3-44c0-9eb0-d02d41d65997\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" Jul 17 22:29:34.434443 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.434292 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1112492-cde3-44c0-9eb0-d02d41d65997-openshift-service-ca\") pod \"maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr\" (UID: \"e1112492-cde3-44c0-9eb0-d02d41d65997\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" Jul 17 22:29:34.434443 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.434368 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1112492-cde3-44c0-9eb0-d02d41d65997-maas-api-tls\") pod \"maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr\" (UID: \"e1112492-cde3-44c0-9eb0-d02d41d65997\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" Jul 17 22:29:34.434945 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.434924 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1112492-cde3-44c0-9eb0-d02d41d65997-openshift-service-ca\") pod \"maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr\" (UID: \"e1112492-cde3-44c0-9eb0-d02d41d65997\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" Jul 17 22:29:34.436952 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.436931 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1112492-cde3-44c0-9eb0-d02d41d65997-maas-api-tls\") pod \"maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr\" (UID: \"e1112492-cde3-44c0-9eb0-d02d41d65997\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" Jul 17 22:29:34.443143 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.443116 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqs57\" (UniqueName: \"kubernetes.io/projected/e1112492-cde3-44c0-9eb0-d02d41d65997-kube-api-access-bqs57\") pod \"maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr\" (UID: \"e1112492-cde3-44c0-9eb0-d02d41d65997\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" Jul 17 22:29:34.580183 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.580099 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" Jul 17 22:29:34.927225 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.927199 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr"] Jul 17 22:29:34.929792 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:29:34.929767 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1112492_cde3_44c0_9eb0_d02d41d65997.slice/crio-8e140bf62109f2ceae037792ffe032c6566a5b91b344b23bb5e28ecbb31469c3 WatchSource:0}: Error finding container 8e140bf62109f2ceae037792ffe032c6566a5b91b344b23bb5e28ecbb31469c3: Status 404 returned error can't find the container with id 8e140bf62109f2ceae037792ffe032c6566a5b91b344b23bb5e28ecbb31469c3 Jul 17 22:29:34.931720 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:34.931700 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 22:29:35.274004 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:35.273969 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" event={"ID":"e1112492-cde3-44c0-9eb0-d02d41d65997","Type":"ContainerStarted","Data":"8e140bf62109f2ceae037792ffe032c6566a5b91b344b23bb5e28ecbb31469c3"} Jul 17 22:29:36.280314 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:36.280282 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" event={"ID":"e1112492-cde3-44c0-9eb0-d02d41d65997","Type":"ContainerStarted","Data":"1e6f257bed2d6b4b0ec502d87dde023904d591290173b38d762a3089549f4bd9"} Jul 17 22:29:36.280748 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:36.280420 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" Jul 17 22:29:36.312568 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:36.312510 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" podStartSLOduration=1.987475178 podStartE2EDuration="2.31246326s" totalImagesPullingTime="324.988082ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:29:36.31176315 +0000 UTC m=+2613.171986654" watchObservedRunningTime="2026-07-17 22:29:36.31246326 +0000 UTC m=+2613.172686763" Jul 17 22:29:42.291630 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:29:42.291599 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" Jul 17 22:30:32.878180 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:32.878147 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr"] Jul 17 22:30:32.878671 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:32.878425 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" podUID="e1112492-cde3-44c0-9eb0-d02d41d65997" containerName="maas-api" containerID="cri-o://1e6f257bed2d6b4b0ec502d87dde023904d591290173b38d762a3089549f4bd9" gracePeriod=30 Jul 17 22:30:33.133297 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.133241 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" Jul 17 22:30:33.273635 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.273596 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1112492-cde3-44c0-9eb0-d02d41d65997-maas-api-tls\") pod \"e1112492-cde3-44c0-9eb0-d02d41d65997\" (UID: \"e1112492-cde3-44c0-9eb0-d02d41d65997\") " Jul 17 22:30:33.273829 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.273699 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1112492-cde3-44c0-9eb0-d02d41d65997-openshift-service-ca\") pod \"e1112492-cde3-44c0-9eb0-d02d41d65997\" (UID: \"e1112492-cde3-44c0-9eb0-d02d41d65997\") " Jul 17 22:30:33.273829 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.273761 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqs57\" (UniqueName: \"kubernetes.io/projected/e1112492-cde3-44c0-9eb0-d02d41d65997-kube-api-access-bqs57\") pod \"e1112492-cde3-44c0-9eb0-d02d41d65997\" (UID: \"e1112492-cde3-44c0-9eb0-d02d41d65997\") " Jul 17 22:30:33.274095 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.274069 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1112492-cde3-44c0-9eb0-d02d41d65997-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e1112492-cde3-44c0-9eb0-d02d41d65997" (UID: "e1112492-cde3-44c0-9eb0-d02d41d65997"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:30:33.275953 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.275933 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1112492-cde3-44c0-9eb0-d02d41d65997-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e1112492-cde3-44c0-9eb0-d02d41d65997" (UID: "e1112492-cde3-44c0-9eb0-d02d41d65997"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:30:33.276044 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.275947 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1112492-cde3-44c0-9eb0-d02d41d65997-kube-api-access-bqs57" (OuterVolumeSpecName: "kube-api-access-bqs57") pod "e1112492-cde3-44c0-9eb0-d02d41d65997" (UID: "e1112492-cde3-44c0-9eb0-d02d41d65997"). InnerVolumeSpecName "kube-api-access-bqs57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:30:33.375046 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.375018 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqs57\" (UniqueName: \"kubernetes.io/projected/e1112492-cde3-44c0-9eb0-d02d41d65997-kube-api-access-bqs57\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:30:33.375046 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.375042 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1112492-cde3-44c0-9eb0-d02d41d65997-maas-api-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:30:33.375046 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.375052 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1112492-cde3-44c0-9eb0-d02d41d65997-openshift-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:30:33.583781 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.583748 2577 generic.go:356] "Generic (PLEG): container finished" podID="e1112492-cde3-44c0-9eb0-d02d41d65997" containerID="1e6f257bed2d6b4b0ec502d87dde023904d591290173b38d762a3089549f4bd9" exitCode=0 Jul 17 22:30:33.583995 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.583796 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" Jul 17 22:30:33.583995 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.583842 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" event={"ID":"e1112492-cde3-44c0-9eb0-d02d41d65997","Type":"ContainerDied","Data":"1e6f257bed2d6b4b0ec502d87dde023904d591290173b38d762a3089549f4bd9"} Jul 17 22:30:33.583995 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.583907 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr" event={"ID":"e1112492-cde3-44c0-9eb0-d02d41d65997","Type":"ContainerDied","Data":"8e140bf62109f2ceae037792ffe032c6566a5b91b344b23bb5e28ecbb31469c3"} Jul 17 22:30:33.583995 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.583939 2577 scope.go:117] "RemoveContainer" containerID="1e6f257bed2d6b4b0ec502d87dde023904d591290173b38d762a3089549f4bd9" Jul 17 22:30:33.594989 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.594971 2577 scope.go:117] "RemoveContainer" containerID="1e6f257bed2d6b4b0ec502d87dde023904d591290173b38d762a3089549f4bd9" Jul 17 22:30:33.595271 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:30:33.595253 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e6f257bed2d6b4b0ec502d87dde023904d591290173b38d762a3089549f4bd9\": container with ID starting with 1e6f257bed2d6b4b0ec502d87dde023904d591290173b38d762a3089549f4bd9 not found: ID does not exist" containerID="1e6f257bed2d6b4b0ec502d87dde023904d591290173b38d762a3089549f4bd9" Jul 17 22:30:33.595327 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.595279 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e6f257bed2d6b4b0ec502d87dde023904d591290173b38d762a3089549f4bd9"} err="failed to get container status \"1e6f257bed2d6b4b0ec502d87dde023904d591290173b38d762a3089549f4bd9\": rpc error: code = NotFound desc = could not find container \"1e6f257bed2d6b4b0ec502d87dde023904d591290173b38d762a3089549f4bd9\": container with ID starting with 1e6f257bed2d6b4b0ec502d87dde023904d591290173b38d762a3089549f4bd9 not found: ID does not exist" Jul 17 22:30:33.605408 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.605382 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr"] Jul 17 22:30:33.610862 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.610840 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-895e50b8-67f84cb76-2fpfr"] Jul 17 22:30:33.700451 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:33.700412 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1112492-cde3-44c0-9eb0-d02d41d65997" path="/var/lib/kubelet/pods/e1112492-cde3-44c0-9eb0-d02d41d65997/volumes" Jul 17 22:30:46.404617 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:46.404585 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f"] Jul 17 22:30:46.405029 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:46.404989 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerName="maas-api" Jul 17 22:30:46.405029 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:46.405000 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="815e3f00-fe5f-44ae-a4d9-05facabccfa1" containerName="maas-api" Jul 17 22:30:46.405029 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:46.405018 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1112492-cde3-44c0-9eb0-d02d41d65997" containerName="maas-api" Jul 17 22:30:46.405029 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:46.405025 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1112492-cde3-44c0-9eb0-d02d41d65997" containerName="maas-api" Jul 17 22:30:46.405185 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:46.405097 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1112492-cde3-44c0-9eb0-d02d41d65997" containerName="maas-api" Jul 17 22:30:46.408312 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:46.408296 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f" Jul 17 22:30:46.412393 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:46.412373 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\"" Jul 17 22:30:46.428392 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:46.428371 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f"] Jul 17 22:30:46.492801 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:46.492769 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e46124a-983a-475f-bb8a-690ba18750d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-rss8f\" (UID: \"9e46124a-983a-475f-bb8a-690ba18750d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f" Jul 17 22:30:46.593313 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:46.593271 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e46124a-983a-475f-bb8a-690ba18750d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-rss8f\" (UID: \"9e46124a-983a-475f-bb8a-690ba18750d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f" Jul 17 22:30:46.593976 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:46.593954 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e46124a-983a-475f-bb8a-690ba18750d8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-09e18326-rss8f\" (UID: \"9e46124a-983a-475f-bb8a-690ba18750d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f" Jul 17 22:30:46.719056 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:46.719024 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f" Jul 17 22:30:46.860012 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:46.859966 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f"] Jul 17 22:30:46.862414 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:30:46.862388 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e46124a_983a_475f_bb8a_690ba18750d8.slice/crio-54b5e6207e9018ea8c9cbfeda9883f3fbe9efdf95c3bd76a19d2f0be16285fa0 WatchSource:0}: Error finding container 54b5e6207e9018ea8c9cbfeda9883f3fbe9efdf95c3bd76a19d2f0be16285fa0: Status 404 returned error can't find the container with id 54b5e6207e9018ea8c9cbfeda9883f3fbe9efdf95c3bd76a19d2f0be16285fa0 Jul 17 22:30:47.657855 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:47.657819 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f" event={"ID":"9e46124a-983a-475f-bb8a-690ba18750d8","Type":"ContainerStarted","Data":"54b5e6207e9018ea8c9cbfeda9883f3fbe9efdf95c3bd76a19d2f0be16285fa0"} Jul 17 22:30:49.670967 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:49.670864 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f" event={"ID":"9e46124a-983a-475f-bb8a-690ba18750d8","Type":"ContainerStarted","Data":"7c7313867ad144e61c0a99889e039c134e1464e41e16832ef21cf9c61933b53b"} Jul 17 22:30:49.694961 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:49.694910 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f" podStartSLOduration=1.506970575 podStartE2EDuration="3.694893821s" totalImagesPullingTime="2.187923246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:30:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:30:49.692622477 +0000 UTC m=+2686.552846000" watchObservedRunningTime="2026-07-17 22:30:49.694893821 +0000 UTC m=+2686.555117332" Jul 17 22:30:50.678148 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:50.678112 2577 generic.go:356] "Generic (PLEG): container finished" podID="9e46124a-983a-475f-bb8a-690ba18750d8" containerID="7c7313867ad144e61c0a99889e039c134e1464e41e16832ef21cf9c61933b53b" exitCode=6 Jul 17 22:30:50.678572 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:50.678156 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f" event={"ID":"9e46124a-983a-475f-bb8a-690ba18750d8","Type":"ContainerDied","Data":"7c7313867ad144e61c0a99889e039c134e1464e41e16832ef21cf9c61933b53b"} Jul 17 22:30:50.678572 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:50.678390 2577 scope.go:117] "RemoveContainer" containerID="7c7313867ad144e61c0a99889e039c134e1464e41e16832ef21cf9c61933b53b" Jul 17 22:30:51.685258 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:51.685163 2577 generic.go:356] "Generic (PLEG): container finished" podID="9e46124a-983a-475f-bb8a-690ba18750d8" containerID="b5973f60a03d6a9018299a748b9a0c3f9df19d0826d84f5a16a0662ade268531" exitCode=6 Jul 17 22:30:51.685258 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:51.685244 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f" event={"ID":"9e46124a-983a-475f-bb8a-690ba18750d8","Type":"ContainerDied","Data":"b5973f60a03d6a9018299a748b9a0c3f9df19d0826d84f5a16a0662ade268531"} Jul 17 22:30:51.685790 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:51.685293 2577 scope.go:117] "RemoveContainer" containerID="7c7313867ad144e61c0a99889e039c134e1464e41e16832ef21cf9c61933b53b" Jul 17 22:30:51.685790 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:51.685581 2577 scope.go:117] "RemoveContainer" containerID="b5973f60a03d6a9018299a748b9a0c3f9df19d0826d84f5a16a0662ade268531" Jul 17 22:30:51.685896 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:30:51.685834 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-rss8f_odh-ai-gateway-infra(9e46124a-983a-475f-bb8a-690ba18750d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f" podUID="9e46124a-983a-475f-bb8a-690ba18750d8" Jul 17 22:30:52.692227 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:52.692202 2577 scope.go:117] "RemoveContainer" containerID="b5973f60a03d6a9018299a748b9a0c3f9df19d0826d84f5a16a0662ade268531" Jul 17 22:30:52.692640 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:30:52.692394 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-rss8f_odh-ai-gateway-infra(9e46124a-983a-475f-bb8a-690ba18750d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f" podUID="9e46124a-983a-475f-bb8a-690ba18750d8" Jul 17 22:30:58.153809 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:58.153732 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4"] Jul 17 22:30:58.158770 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:58.158750 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4" Jul 17 22:30:58.165530 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:58.165467 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4"] Jul 17 22:30:58.198723 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:58.198692 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11bcd503-3da5-4e11-ae51-fafc1b1c8f45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4\" (UID: \"11bcd503-3da5-4e11-ae51-fafc1b1c8f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4" Jul 17 22:30:58.299294 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:58.299259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11bcd503-3da5-4e11-ae51-fafc1b1c8f45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4\" (UID: \"11bcd503-3da5-4e11-ae51-fafc1b1c8f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4" Jul 17 22:30:58.299959 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:58.299931 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11bcd503-3da5-4e11-ae51-fafc1b1c8f45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4\" (UID: \"11bcd503-3da5-4e11-ae51-fafc1b1c8f45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4" Jul 17 22:30:58.470663 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:58.470629 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4" Jul 17 22:30:58.608679 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:58.608652 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4"] Jul 17 22:30:58.610814 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:30:58.610790 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11bcd503_3da5_4e11_ae51_fafc1b1c8f45.slice/crio-fc46ce86e7288eb091b3ea2bcd5db660bce57255627935bfe86fafbaa9b3558c WatchSource:0}: Error finding container fc46ce86e7288eb091b3ea2bcd5db660bce57255627935bfe86fafbaa9b3558c: Status 404 returned error can't find the container with id fc46ce86e7288eb091b3ea2bcd5db660bce57255627935bfe86fafbaa9b3558c Jul 17 22:30:58.723257 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:58.723172 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4" event={"ID":"11bcd503-3da5-4e11-ae51-fafc1b1c8f45","Type":"ContainerStarted","Data":"fc46ce86e7288eb091b3ea2bcd5db660bce57255627935bfe86fafbaa9b3558c"} Jul 17 22:30:59.729871 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:59.729833 2577 generic.go:356] "Generic (PLEG): container finished" podID="11bcd503-3da5-4e11-ae51-fafc1b1c8f45" containerID="cb29edd3f5a1710d60227b5a56a1f23e9a208dfece7e72545f4aa755e4c1b2be" exitCode=6 Jul 17 22:30:59.730360 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:59.729880 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4" event={"ID":"11bcd503-3da5-4e11-ae51-fafc1b1c8f45","Type":"ContainerDied","Data":"cb29edd3f5a1710d60227b5a56a1f23e9a208dfece7e72545f4aa755e4c1b2be"} Jul 17 22:30:59.730360 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:30:59.730101 2577 scope.go:117] "RemoveContainer" containerID="cb29edd3f5a1710d60227b5a56a1f23e9a208dfece7e72545f4aa755e4c1b2be" Jul 17 22:31:00.737047 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:00.737011 2577 generic.go:356] "Generic (PLEG): container finished" podID="11bcd503-3da5-4e11-ae51-fafc1b1c8f45" containerID="b638113e0fbc86dba1bcf4b339371e4f0263fa4eccdc0e083eb3514eff49d883" exitCode=6 Jul 17 22:31:00.737508 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:00.737095 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4" event={"ID":"11bcd503-3da5-4e11-ae51-fafc1b1c8f45","Type":"ContainerDied","Data":"b638113e0fbc86dba1bcf4b339371e4f0263fa4eccdc0e083eb3514eff49d883"} Jul 17 22:31:00.737508 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:00.737152 2577 scope.go:117] "RemoveContainer" containerID="cb29edd3f5a1710d60227b5a56a1f23e9a208dfece7e72545f4aa755e4c1b2be" Jul 17 22:31:00.737508 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:00.737297 2577 scope.go:117] "RemoveContainer" containerID="b638113e0fbc86dba1bcf4b339371e4f0263fa4eccdc0e083eb3514eff49d883" Jul 17 22:31:00.737688 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:31:00.737538 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4_odh-ai-gateway-infra(11bcd503-3da5-4e11-ae51-fafc1b1c8f45)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4" podUID="11bcd503-3da5-4e11-ae51-fafc1b1c8f45" Jul 17 22:31:01.744174 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:01.744147 2577 scope.go:117] "RemoveContainer" containerID="b638113e0fbc86dba1bcf4b339371e4f0263fa4eccdc0e083eb3514eff49d883" Jul 17 22:31:01.744594 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:31:01.744346 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4_odh-ai-gateway-infra(11bcd503-3da5-4e11-ae51-fafc1b1c8f45)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4" podUID="11bcd503-3da5-4e11-ae51-fafc1b1c8f45" Jul 17 22:31:07.695078 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:07.695043 2577 scope.go:117] "RemoveContainer" containerID="b5973f60a03d6a9018299a748b9a0c3f9df19d0826d84f5a16a0662ade268531" Jul 17 22:31:08.783811 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:08.783778 2577 generic.go:356] "Generic (PLEG): container finished" podID="9e46124a-983a-475f-bb8a-690ba18750d8" containerID="ee682f79748029abbc13626e2cbc7fa2458ee7ef040fa41bb59fe0152830bd12" exitCode=6 Jul 17 22:31:08.784294 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:08.783849 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f" event={"ID":"9e46124a-983a-475f-bb8a-690ba18750d8","Type":"ContainerDied","Data":"ee682f79748029abbc13626e2cbc7fa2458ee7ef040fa41bb59fe0152830bd12"} Jul 17 22:31:08.784294 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:08.783908 2577 scope.go:117] "RemoveContainer" containerID="b5973f60a03d6a9018299a748b9a0c3f9df19d0826d84f5a16a0662ade268531" Jul 17 22:31:08.784294 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:08.784224 2577 scope.go:117] "RemoveContainer" containerID="ee682f79748029abbc13626e2cbc7fa2458ee7ef040fa41bb59fe0152830bd12" Jul 17 22:31:08.784544 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:31:08.784481 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-09e18326-rss8f_odh-ai-gateway-infra(9e46124a-983a-475f-bb8a-690ba18750d8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f" podUID="9e46124a-983a-475f-bb8a-690ba18750d8" Jul 17 22:31:09.814274 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:09.814233 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f"] Jul 17 22:31:09.957196 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:09.957170 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f" Jul 17 22:31:10.117484 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:10.117388 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e46124a-983a-475f-bb8a-690ba18750d8-maas-api-service-ca\") pod \"9e46124a-983a-475f-bb8a-690ba18750d8\" (UID: \"9e46124a-983a-475f-bb8a-690ba18750d8\") " Jul 17 22:31:10.117784 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:10.117762 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e46124a-983a-475f-bb8a-690ba18750d8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9e46124a-983a-475f-bb8a-690ba18750d8" (UID: "9e46124a-983a-475f-bb8a-690ba18750d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:31:10.218723 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:10.218684 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e46124a-983a-475f-bb8a-690ba18750d8-maas-api-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:31:10.797873 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:10.797845 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f" Jul 17 22:31:10.798060 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:10.797894 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f" event={"ID":"9e46124a-983a-475f-bb8a-690ba18750d8","Type":"ContainerDied","Data":"54b5e6207e9018ea8c9cbfeda9883f3fbe9efdf95c3bd76a19d2f0be16285fa0"} Jul 17 22:31:10.798060 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:10.797952 2577 scope.go:117] "RemoveContainer" containerID="ee682f79748029abbc13626e2cbc7fa2458ee7ef040fa41bb59fe0152830bd12" Jul 17 22:31:10.819853 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:10.819824 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f"] Jul 17 22:31:10.827586 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:10.827560 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-09e18326-rss8f"] Jul 17 22:31:11.700697 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:11.700665 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e46124a-983a-475f-bb8a-690ba18750d8" path="/var/lib/kubelet/pods/9e46124a-983a-475f-bb8a-690ba18750d8/volumes" Jul 17 22:31:16.694751 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:16.694716 2577 scope.go:117] "RemoveContainer" containerID="b638113e0fbc86dba1bcf4b339371e4f0263fa4eccdc0e083eb3514eff49d883" Jul 17 22:31:17.838420 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:17.838379 2577 generic.go:356] "Generic (PLEG): container finished" podID="11bcd503-3da5-4e11-ae51-fafc1b1c8f45" containerID="dc41c40c56d94710e85ca78b8d65ec5e73e90b4cc35a5ea1d185c41dba6974a8" exitCode=6 Jul 17 22:31:17.838940 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:17.838454 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4" event={"ID":"11bcd503-3da5-4e11-ae51-fafc1b1c8f45","Type":"ContainerDied","Data":"dc41c40c56d94710e85ca78b8d65ec5e73e90b4cc35a5ea1d185c41dba6974a8"} Jul 17 22:31:17.838940 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:17.838530 2577 scope.go:117] "RemoveContainer" containerID="b638113e0fbc86dba1bcf4b339371e4f0263fa4eccdc0e083eb3514eff49d883" Jul 17 22:31:17.838940 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:17.838891 2577 scope.go:117] "RemoveContainer" containerID="dc41c40c56d94710e85ca78b8d65ec5e73e90b4cc35a5ea1d185c41dba6974a8" Jul 17 22:31:17.839179 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:31:17.839153 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4_odh-ai-gateway-infra(11bcd503-3da5-4e11-ae51-fafc1b1c8f45)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4" podUID="11bcd503-3da5-4e11-ae51-fafc1b1c8f45" Jul 17 22:31:18.866184 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:18.866152 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4"] Jul 17 22:31:19.011204 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:19.011180 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4" Jul 17 22:31:19.099612 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:19.099572 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11bcd503-3da5-4e11-ae51-fafc1b1c8f45-maas-api-service-ca\") pod \"11bcd503-3da5-4e11-ae51-fafc1b1c8f45\" (UID: \"11bcd503-3da5-4e11-ae51-fafc1b1c8f45\") " Jul 17 22:31:19.099931 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:19.099906 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11bcd503-3da5-4e11-ae51-fafc1b1c8f45-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "11bcd503-3da5-4e11-ae51-fafc1b1c8f45" (UID: "11bcd503-3da5-4e11-ae51-fafc1b1c8f45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:31:19.099997 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:19.099984 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11bcd503-3da5-4e11-ae51-fafc1b1c8f45-maas-api-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:31:19.853378 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:19.853339 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4" event={"ID":"11bcd503-3da5-4e11-ae51-fafc1b1c8f45","Type":"ContainerDied","Data":"fc46ce86e7288eb091b3ea2bcd5db660bce57255627935bfe86fafbaa9b3558c"} Jul 17 22:31:19.853378 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:19.853366 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4" Jul 17 22:31:19.853628 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:19.853415 2577 scope.go:117] "RemoveContainer" containerID="dc41c40c56d94710e85ca78b8d65ec5e73e90b4cc35a5ea1d185c41dba6974a8" Jul 17 22:31:19.868912 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:19.868888 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4"] Jul 17 22:31:19.876573 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:19.876551 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-btgm4"] Jul 17 22:31:21.700053 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:21.700022 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11bcd503-3da5-4e11-ae51-fafc1b1c8f45" path="/var/lib/kubelet/pods/11bcd503-3da5-4e11-ae51-fafc1b1c8f45/volumes" Jul 17 22:31:24.878545 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:24.878518 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7"] Jul 17 22:31:24.879109 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:24.879091 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11bcd503-3da5-4e11-ae51-fafc1b1c8f45" containerName="revoke-keys" Jul 17 22:31:24.879200 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:24.879113 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="11bcd503-3da5-4e11-ae51-fafc1b1c8f45" containerName="revoke-keys" Jul 17 22:31:24.879200 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:24.879130 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e46124a-983a-475f-bb8a-690ba18750d8" containerName="revoke-keys" Jul 17 22:31:24.879200 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:24.879138 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e46124a-983a-475f-bb8a-690ba18750d8" containerName="revoke-keys" Jul 17 22:31:24.879200 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:24.879157 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e46124a-983a-475f-bb8a-690ba18750d8" containerName="revoke-keys" Jul 17 22:31:24.879200 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:24.879165 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e46124a-983a-475f-bb8a-690ba18750d8" containerName="revoke-keys" Jul 17 22:31:24.879200 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:24.879183 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e46124a-983a-475f-bb8a-690ba18750d8" containerName="revoke-keys" Jul 17 22:31:24.879200 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:24.879191 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e46124a-983a-475f-bb8a-690ba18750d8" containerName="revoke-keys" Jul 17 22:31:24.879624 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:24.879210 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11bcd503-3da5-4e11-ae51-fafc1b1c8f45" containerName="revoke-keys" Jul 17 22:31:24.879624 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:24.879218 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="11bcd503-3da5-4e11-ae51-fafc1b1c8f45" containerName="revoke-keys" Jul 17 22:31:24.879624 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:24.879327 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e46124a-983a-475f-bb8a-690ba18750d8" containerName="revoke-keys" Jul 17 22:31:24.879624 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:24.879339 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="11bcd503-3da5-4e11-ae51-fafc1b1c8f45" containerName="revoke-keys" Jul 17 22:31:24.879624 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:24.879348 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="11bcd503-3da5-4e11-ae51-fafc1b1c8f45" containerName="revoke-keys" Jul 17 22:31:24.879624 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:24.879357 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e46124a-983a-475f-bb8a-690ba18750d8" containerName="revoke-keys" Jul 17 22:31:24.885699 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:24.885677 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7" Jul 17 22:31:24.889838 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:24.889821 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\"" Jul 17 22:31:24.902873 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:24.902840 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7"] Jul 17 22:31:24.951325 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:24.951291 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6889103d-8551-48b5-88b2-6d036d6521b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7\" (UID: \"6889103d-8551-48b5-88b2-6d036d6521b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7" Jul 17 22:31:25.052594 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:25.052555 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6889103d-8551-48b5-88b2-6d036d6521b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7\" (UID: \"6889103d-8551-48b5-88b2-6d036d6521b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7" Jul 17 22:31:25.053226 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:25.053202 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6889103d-8551-48b5-88b2-6d036d6521b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7\" (UID: \"6889103d-8551-48b5-88b2-6d036d6521b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7" Jul 17 22:31:25.195696 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:25.195661 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7" Jul 17 22:31:25.338477 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:25.338453 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7"] Jul 17 22:31:25.340511 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:31:25.340466 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6889103d_8551_48b5_88b2_6d036d6521b2.slice/crio-992474846cf093b83f1db86726ebd470d0118e79fd0d2f99260ea3e43da5be5e WatchSource:0}: Error finding container 992474846cf093b83f1db86726ebd470d0118e79fd0d2f99260ea3e43da5be5e: Status 404 returned error can't find the container with id 992474846cf093b83f1db86726ebd470d0118e79fd0d2f99260ea3e43da5be5e Jul 17 22:31:25.888522 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:25.888405 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7" event={"ID":"6889103d-8551-48b5-88b2-6d036d6521b2","Type":"ContainerStarted","Data":"375a3c59240644a22af9617bfa5cc296d7b3d7fd9668d4d4d5544368811c482e"} Jul 17 22:31:25.888522 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:25.888466 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7" event={"ID":"6889103d-8551-48b5-88b2-6d036d6521b2","Type":"ContainerStarted","Data":"992474846cf093b83f1db86726ebd470d0118e79fd0d2f99260ea3e43da5be5e"} Jul 17 22:31:25.905367 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:25.905315 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7" podStartSLOduration=1.905299103 podStartE2EDuration="1.905299103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:31:25.903457792 +0000 UTC m=+2722.763681293" watchObservedRunningTime="2026-07-17 22:31:25.905299103 +0000 UTC m=+2722.765522635" Jul 17 22:31:26.895409 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:26.895375 2577 generic.go:356] "Generic (PLEG): container finished" podID="6889103d-8551-48b5-88b2-6d036d6521b2" containerID="375a3c59240644a22af9617bfa5cc296d7b3d7fd9668d4d4d5544368811c482e" exitCode=6 Jul 17 22:31:26.895409 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:26.895414 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7" event={"ID":"6889103d-8551-48b5-88b2-6d036d6521b2","Type":"ContainerDied","Data":"375a3c59240644a22af9617bfa5cc296d7b3d7fd9668d4d4d5544368811c482e"} Jul 17 22:31:26.895855 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:26.895613 2577 scope.go:117] "RemoveContainer" containerID="375a3c59240644a22af9617bfa5cc296d7b3d7fd9668d4d4d5544368811c482e" Jul 17 22:31:27.903333 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:27.903240 2577 generic.go:356] "Generic (PLEG): container finished" podID="6889103d-8551-48b5-88b2-6d036d6521b2" containerID="ae76943bc840a07f5527605b3ceadd473e3811c5d2684fdae0f2d30df7c2b76e" exitCode=6 Jul 17 22:31:27.903805 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:27.903330 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7" event={"ID":"6889103d-8551-48b5-88b2-6d036d6521b2","Type":"ContainerDied","Data":"ae76943bc840a07f5527605b3ceadd473e3811c5d2684fdae0f2d30df7c2b76e"} Jul 17 22:31:27.903805 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:27.903389 2577 scope.go:117] "RemoveContainer" containerID="375a3c59240644a22af9617bfa5cc296d7b3d7fd9668d4d4d5544368811c482e" Jul 17 22:31:27.903805 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:27.903613 2577 scope.go:117] "RemoveContainer" containerID="ae76943bc840a07f5527605b3ceadd473e3811c5d2684fdae0f2d30df7c2b76e" Jul 17 22:31:27.903929 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:31:27.903817 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7_odh-ai-gateway-infra(6889103d-8551-48b5-88b2-6d036d6521b2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7" podUID="6889103d-8551-48b5-88b2-6d036d6521b2" Jul 17 22:31:28.910989 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:28.910959 2577 scope.go:117] "RemoveContainer" containerID="ae76943bc840a07f5527605b3ceadd473e3811c5d2684fdae0f2d30df7c2b76e" Jul 17 22:31:28.911380 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:31:28.911163 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7_odh-ai-gateway-infra(6889103d-8551-48b5-88b2-6d036d6521b2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7" podUID="6889103d-8551-48b5-88b2-6d036d6521b2" Jul 17 22:31:40.694657 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:40.694622 2577 scope.go:117] "RemoveContainer" containerID="ae76943bc840a07f5527605b3ceadd473e3811c5d2684fdae0f2d30df7c2b76e" Jul 17 22:31:41.984735 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:41.984704 2577 generic.go:356] "Generic (PLEG): container finished" podID="6889103d-8551-48b5-88b2-6d036d6521b2" containerID="cca11c9c199fb6a0b745729dc9f4b7d7c454262b5558e9f16d445cf51b4161b1" exitCode=6 Jul 17 22:31:41.985143 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:41.984789 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7" event={"ID":"6889103d-8551-48b5-88b2-6d036d6521b2","Type":"ContainerDied","Data":"cca11c9c199fb6a0b745729dc9f4b7d7c454262b5558e9f16d445cf51b4161b1"} Jul 17 22:31:41.985143 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:41.984864 2577 scope.go:117] "RemoveContainer" containerID="ae76943bc840a07f5527605b3ceadd473e3811c5d2684fdae0f2d30df7c2b76e" Jul 17 22:31:41.985261 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:41.985197 2577 scope.go:117] "RemoveContainer" containerID="cca11c9c199fb6a0b745729dc9f4b7d7c454262b5558e9f16d445cf51b4161b1" Jul 17 22:31:41.985430 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:31:41.985406 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7_odh-ai-gateway-infra(6889103d-8551-48b5-88b2-6d036d6521b2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7" podUID="6889103d-8551-48b5-88b2-6d036d6521b2" Jul 17 22:31:43.013456 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:43.013424 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7"] Jul 17 22:31:43.159458 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:43.159435 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7" Jul 17 22:31:43.231036 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:43.231001 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6889103d-8551-48b5-88b2-6d036d6521b2-maas-api-service-ca\") pod \"6889103d-8551-48b5-88b2-6d036d6521b2\" (UID: \"6889103d-8551-48b5-88b2-6d036d6521b2\") " Jul 17 22:31:43.231343 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:43.231320 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6889103d-8551-48b5-88b2-6d036d6521b2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6889103d-8551-48b5-88b2-6d036d6521b2" (UID: "6889103d-8551-48b5-88b2-6d036d6521b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:31:43.332272 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:43.332192 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6889103d-8551-48b5-88b2-6d036d6521b2-maas-api-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:31:44.000931 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:44.000829 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7" event={"ID":"6889103d-8551-48b5-88b2-6d036d6521b2","Type":"ContainerDied","Data":"992474846cf093b83f1db86726ebd470d0118e79fd0d2f99260ea3e43da5be5e"} Jul 17 22:31:44.001109 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:44.000946 2577 scope.go:117] "RemoveContainer" containerID="cca11c9c199fb6a0b745729dc9f4b7d7c454262b5558e9f16d445cf51b4161b1" Jul 17 22:31:44.001169 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:44.001120 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7" Jul 17 22:31:44.023764 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:44.023737 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7"] Jul 17 22:31:44.030546 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:44.030518 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-45495ac6-jnsc7"] Jul 17 22:31:45.701773 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:31:45.701737 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6889103d-8551-48b5-88b2-6d036d6521b2" path="/var/lib/kubelet/pods/6889103d-8551-48b5-88b2-6d036d6521b2/volumes" Jul 17 22:32:30.699090 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.699007 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg"] Jul 17 22:32:30.699557 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.699472 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6889103d-8551-48b5-88b2-6d036d6521b2" containerName="revoke-keys" Jul 17 22:32:30.699557 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.699483 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6889103d-8551-48b5-88b2-6d036d6521b2" containerName="revoke-keys" Jul 17 22:32:30.699557 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.699508 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11bcd503-3da5-4e11-ae51-fafc1b1c8f45" containerName="revoke-keys" Jul 17 22:32:30.699557 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.699514 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="11bcd503-3da5-4e11-ae51-fafc1b1c8f45" containerName="revoke-keys" Jul 17 22:32:30.699557 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.699519 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6889103d-8551-48b5-88b2-6d036d6521b2" containerName="revoke-keys" Jul 17 22:32:30.699557 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.699525 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6889103d-8551-48b5-88b2-6d036d6521b2" containerName="revoke-keys" Jul 17 22:32:30.699759 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.699599 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6889103d-8551-48b5-88b2-6d036d6521b2" containerName="revoke-keys" Jul 17 22:32:30.699759 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.699610 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="11bcd503-3da5-4e11-ae51-fafc1b1c8f45" containerName="revoke-keys" Jul 17 22:32:30.699759 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.699617 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e46124a-983a-475f-bb8a-690ba18750d8" containerName="revoke-keys" Jul 17 22:32:30.699759 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.699624 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6889103d-8551-48b5-88b2-6d036d6521b2" containerName="revoke-keys" Jul 17 22:32:30.702969 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.702948 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" Jul 17 22:32:30.712981 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.712958 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-11aecf-serving-cert\"" Jul 17 22:32:30.731311 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.731282 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg"] Jul 17 22:32:30.870428 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.870397 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/299e1f73-1041-4eb4-8a07-4b76dfcae186-openshift-service-ca\") pod \"maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg\" (UID: \"299e1f73-1041-4eb4-8a07-4b76dfcae186\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" Jul 17 22:32:30.870624 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.870455 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmj98\" (UniqueName: \"kubernetes.io/projected/299e1f73-1041-4eb4-8a07-4b76dfcae186-kube-api-access-fmj98\") pod \"maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg\" (UID: \"299e1f73-1041-4eb4-8a07-4b76dfcae186\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" Jul 17 22:32:30.870681 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.870624 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/299e1f73-1041-4eb4-8a07-4b76dfcae186-maas-api-tls\") pod \"maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg\" (UID: \"299e1f73-1041-4eb4-8a07-4b76dfcae186\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" Jul 17 22:32:30.971297 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.971211 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmj98\" (UniqueName: \"kubernetes.io/projected/299e1f73-1041-4eb4-8a07-4b76dfcae186-kube-api-access-fmj98\") pod \"maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg\" (UID: \"299e1f73-1041-4eb4-8a07-4b76dfcae186\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" Jul 17 22:32:30.971455 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.971314 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/299e1f73-1041-4eb4-8a07-4b76dfcae186-maas-api-tls\") pod \"maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg\" (UID: \"299e1f73-1041-4eb4-8a07-4b76dfcae186\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" Jul 17 22:32:30.971455 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.971365 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/299e1f73-1041-4eb4-8a07-4b76dfcae186-openshift-service-ca\") pod \"maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg\" (UID: \"299e1f73-1041-4eb4-8a07-4b76dfcae186\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" Jul 17 22:32:30.972026 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.972002 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/299e1f73-1041-4eb4-8a07-4b76dfcae186-openshift-service-ca\") pod \"maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg\" (UID: \"299e1f73-1041-4eb4-8a07-4b76dfcae186\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" Jul 17 22:32:30.973988 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.973969 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/299e1f73-1041-4eb4-8a07-4b76dfcae186-maas-api-tls\") pod \"maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg\" (UID: \"299e1f73-1041-4eb4-8a07-4b76dfcae186\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" Jul 17 22:32:30.989792 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:30.989759 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmj98\" (UniqueName: \"kubernetes.io/projected/299e1f73-1041-4eb4-8a07-4b76dfcae186-kube-api-access-fmj98\") pod \"maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg\" (UID: \"299e1f73-1041-4eb4-8a07-4b76dfcae186\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" Jul 17 22:32:31.013748 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:31.013714 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" Jul 17 22:32:31.205525 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:31.205480 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg"] Jul 17 22:32:31.207848 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:32:31.207818 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod299e1f73_1041_4eb4_8a07_4b76dfcae186.slice/crio-31f222902b79dfda647ad6210c1ac19092cda05d93ef12dde9d3b2a869529bbe WatchSource:0}: Error finding container 31f222902b79dfda647ad6210c1ac19092cda05d93ef12dde9d3b2a869529bbe: Status 404 returned error can't find the container with id 31f222902b79dfda647ad6210c1ac19092cda05d93ef12dde9d3b2a869529bbe Jul 17 22:32:31.242936 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:31.242866 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" event={"ID":"299e1f73-1041-4eb4-8a07-4b76dfcae186","Type":"ContainerStarted","Data":"31f222902b79dfda647ad6210c1ac19092cda05d93ef12dde9d3b2a869529bbe"} Jul 17 22:32:32.251018 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:32.250980 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" event={"ID":"299e1f73-1041-4eb4-8a07-4b76dfcae186","Type":"ContainerStarted","Data":"eaadfc9f76c34e15d12fc93fe3b9120fb05e1e32f47ee2eeb66b3b6cc774644f"} Jul 17 22:32:32.251408 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:32.251154 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" Jul 17 22:32:38.262074 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:38.262045 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" Jul 17 22:32:38.314905 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:38.314846 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" podStartSLOduration=7.886965778 podStartE2EDuration="8.31482855s" totalImagesPullingTime="427.862772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:32:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:32:32.2925587 +0000 UTC m=+2789.152782202" watchObservedRunningTime="2026-07-17 22:32:38.31482855 +0000 UTC m=+2795.175052052" Jul 17 22:32:39.549930 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:39.549898 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb"] Jul 17 22:32:39.550327 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:39.550313 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6889103d-8551-48b5-88b2-6d036d6521b2" containerName="revoke-keys" Jul 17 22:32:39.550327 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:39.550328 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6889103d-8551-48b5-88b2-6d036d6521b2" containerName="revoke-keys" Jul 17 22:32:39.550414 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:39.550405 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6889103d-8551-48b5-88b2-6d036d6521b2" containerName="revoke-keys" Jul 17 22:32:39.553663 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:39.553647 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" Jul 17 22:32:39.561207 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:39.560878 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4ccee5-serving-cert\"" Jul 17 22:32:39.584299 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:39.584273 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb"] Jul 17 22:32:39.654709 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:39.654676 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc65781e-8218-4e7a-aa97-f2a35cd84dda-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb\" (UID: \"cc65781e-8218-4e7a-aa97-f2a35cd84dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" Jul 17 22:32:39.654898 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:39.654718 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5j4x\" (UniqueName: \"kubernetes.io/projected/cc65781e-8218-4e7a-aa97-f2a35cd84dda-kube-api-access-r5j4x\") pod \"maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb\" (UID: \"cc65781e-8218-4e7a-aa97-f2a35cd84dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" Jul 17 22:32:39.654898 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:39.654781 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc65781e-8218-4e7a-aa97-f2a35cd84dda-maas-api-tls\") pod \"maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb\" (UID: \"cc65781e-8218-4e7a-aa97-f2a35cd84dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" Jul 17 22:32:39.755174 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:39.755146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5j4x\" (UniqueName: \"kubernetes.io/projected/cc65781e-8218-4e7a-aa97-f2a35cd84dda-kube-api-access-r5j4x\") pod \"maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb\" (UID: \"cc65781e-8218-4e7a-aa97-f2a35cd84dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" Jul 17 22:32:39.755347 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:39.755206 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc65781e-8218-4e7a-aa97-f2a35cd84dda-maas-api-tls\") pod \"maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb\" (UID: \"cc65781e-8218-4e7a-aa97-f2a35cd84dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" Jul 17 22:32:39.755347 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:39.755271 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc65781e-8218-4e7a-aa97-f2a35cd84dda-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb\" (UID: \"cc65781e-8218-4e7a-aa97-f2a35cd84dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" Jul 17 22:32:39.755450 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:32:39.755371 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-serving-cert: secret "maas-api-e2e-inf-b-4ccee5-serving-cert" not found Jul 17 22:32:39.755532 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:32:39.755456 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc65781e-8218-4e7a-aa97-f2a35cd84dda-maas-api-tls podName:cc65781e-8218-4e7a-aa97-f2a35cd84dda nodeName:}" failed. No retries permitted until 2026-07-17 22:32:40.255425944 +0000 UTC m=+2797.115649424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cc65781e-8218-4e7a-aa97-f2a35cd84dda-maas-api-tls") pod "maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" (UID: "cc65781e-8218-4e7a-aa97-f2a35cd84dda") : secret "maas-api-e2e-inf-b-4ccee5-serving-cert" not found Jul 17 22:32:39.755968 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:39.755951 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc65781e-8218-4e7a-aa97-f2a35cd84dda-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb\" (UID: \"cc65781e-8218-4e7a-aa97-f2a35cd84dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" Jul 17 22:32:39.782410 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:39.782384 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5j4x\" (UniqueName: \"kubernetes.io/projected/cc65781e-8218-4e7a-aa97-f2a35cd84dda-kube-api-access-r5j4x\") pod \"maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb\" (UID: \"cc65781e-8218-4e7a-aa97-f2a35cd84dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" Jul 17 22:32:40.264849 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:40.264812 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc65781e-8218-4e7a-aa97-f2a35cd84dda-maas-api-tls\") pod \"maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb\" (UID: \"cc65781e-8218-4e7a-aa97-f2a35cd84dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" Jul 17 22:32:40.267288 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:40.267259 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc65781e-8218-4e7a-aa97-f2a35cd84dda-maas-api-tls\") pod \"maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb\" (UID: \"cc65781e-8218-4e7a-aa97-f2a35cd84dda\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" Jul 17 22:32:40.465458 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:40.465417 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" Jul 17 22:32:40.622917 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:40.622893 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb"] Jul 17 22:32:40.624952 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:32:40.624918 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc65781e_8218_4e7a_aa97_f2a35cd84dda.slice/crio-00cae66a3c23bb891b6b8e2aa6ab1f357e51934a2b039dafdfee23b3e39ea2ca WatchSource:0}: Error finding container 00cae66a3c23bb891b6b8e2aa6ab1f357e51934a2b039dafdfee23b3e39ea2ca: Status 404 returned error can't find the container with id 00cae66a3c23bb891b6b8e2aa6ab1f357e51934a2b039dafdfee23b3e39ea2ca Jul 17 22:32:41.303637 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:41.303542 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" event={"ID":"cc65781e-8218-4e7a-aa97-f2a35cd84dda","Type":"ContainerStarted","Data":"8a6a401713a1ab7e6eaa3b00c36f1d5aa4fc3dc9c693656300e6cffc8d91ee99"} Jul 17 22:32:41.303637 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:41.303596 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" event={"ID":"cc65781e-8218-4e7a-aa97-f2a35cd84dda","Type":"ContainerStarted","Data":"00cae66a3c23bb891b6b8e2aa6ab1f357e51934a2b039dafdfee23b3e39ea2ca"} Jul 17 22:32:41.303836 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:41.303694 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" Jul 17 22:32:41.353546 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:41.353469 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" podStartSLOduration=1.986200264 podStartE2EDuration="2.353450478s" totalImagesPullingTime="367.250214ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:32:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:32:41.344876928 +0000 UTC m=+2798.205100429" watchObservedRunningTime="2026-07-17 22:32:41.353450478 +0000 UTC m=+2798.213674009" Jul 17 22:32:42.855269 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:42.855237 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q"] Jul 17 22:32:42.859178 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:42.859160 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q" Jul 17 22:32:42.861805 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:42.861781 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\"" Jul 17 22:32:42.875526 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:42.875475 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q"] Jul 17 22:32:42.889291 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:42.889264 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4036010d-699e-41ac-8167-031b0dea254c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q\" (UID: \"4036010d-699e-41ac-8167-031b0dea254c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q" Jul 17 22:32:42.990281 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:42.990241 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4036010d-699e-41ac-8167-031b0dea254c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q\" (UID: \"4036010d-699e-41ac-8167-031b0dea254c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q" Jul 17 22:32:42.990891 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:42.990871 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4036010d-699e-41ac-8167-031b0dea254c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q\" (UID: \"4036010d-699e-41ac-8167-031b0dea254c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q" Jul 17 22:32:43.169922 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:43.169835 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q" Jul 17 22:32:43.318792 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:43.318761 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q"] Jul 17 22:32:43.321994 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:32:43.321962 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4036010d_699e_41ac_8167_031b0dea254c.slice/crio-1c815afba30bb2799e22eff8b816829270e8f56a6a7d7837b5988d857a8aa2c3 WatchSource:0}: Error finding container 1c815afba30bb2799e22eff8b816829270e8f56a6a7d7837b5988d857a8aa2c3: Status 404 returned error can't find the container with id 1c815afba30bb2799e22eff8b816829270e8f56a6a7d7837b5988d857a8aa2c3 Jul 17 22:32:44.319580 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:44.319464 2577 generic.go:356] "Generic (PLEG): container finished" podID="4036010d-699e-41ac-8167-031b0dea254c" containerID="d3cbadca9b68147dfc9636dc4387fef0ee6f0fe5b4b35cd2396da1c277f04ecc" exitCode=6 Jul 17 22:32:44.319580 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:44.319532 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q" event={"ID":"4036010d-699e-41ac-8167-031b0dea254c","Type":"ContainerDied","Data":"d3cbadca9b68147dfc9636dc4387fef0ee6f0fe5b4b35cd2396da1c277f04ecc"} Jul 17 22:32:44.320027 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:44.319584 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q" event={"ID":"4036010d-699e-41ac-8167-031b0dea254c","Type":"ContainerStarted","Data":"1c815afba30bb2799e22eff8b816829270e8f56a6a7d7837b5988d857a8aa2c3"} Jul 17 22:32:44.320027 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:44.319755 2577 scope.go:117] "RemoveContainer" containerID="d3cbadca9b68147dfc9636dc4387fef0ee6f0fe5b4b35cd2396da1c277f04ecc" Jul 17 22:32:45.327195 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:45.327159 2577 generic.go:356] "Generic (PLEG): container finished" podID="4036010d-699e-41ac-8167-031b0dea254c" containerID="912083c78f0a8bd6ae2b64aa00d219eb2624ad53697b02bc324419d5937795fa" exitCode=6 Jul 17 22:32:45.327702 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:45.327252 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q" event={"ID":"4036010d-699e-41ac-8167-031b0dea254c","Type":"ContainerDied","Data":"912083c78f0a8bd6ae2b64aa00d219eb2624ad53697b02bc324419d5937795fa"} Jul 17 22:32:45.327702 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:45.327342 2577 scope.go:117] "RemoveContainer" containerID="d3cbadca9b68147dfc9636dc4387fef0ee6f0fe5b4b35cd2396da1c277f04ecc" Jul 17 22:32:45.327702 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:45.327537 2577 scope.go:117] "RemoveContainer" containerID="912083c78f0a8bd6ae2b64aa00d219eb2624ad53697b02bc324419d5937795fa" Jul 17 22:32:45.327870 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:32:45.327810 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q_odh-ai-gateway-infra(4036010d-699e-41ac-8167-031b0dea254c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q" podUID="4036010d-699e-41ac-8167-031b0dea254c" Jul 17 22:32:46.336275 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:46.336246 2577 scope.go:117] "RemoveContainer" containerID="912083c78f0a8bd6ae2b64aa00d219eb2624ad53697b02bc324419d5937795fa" Jul 17 22:32:46.336863 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:32:46.336482 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q_odh-ai-gateway-infra(4036010d-699e-41ac-8167-031b0dea254c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q" podUID="4036010d-699e-41ac-8167-031b0dea254c" Jul 17 22:32:47.315589 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:47.315556 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" Jul 17 22:32:58.335110 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:58.335072 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq"] Jul 17 22:32:58.340130 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:58.340108 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq" Jul 17 22:32:58.366511 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:58.365116 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq"] Jul 17 22:32:58.444187 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:58.444155 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d178eef0-60d1-415b-94ae-8ebb36bf9df5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq\" (UID: \"d178eef0-60d1-415b-94ae-8ebb36bf9df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq" Jul 17 22:32:58.545391 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:58.545353 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d178eef0-60d1-415b-94ae-8ebb36bf9df5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq\" (UID: \"d178eef0-60d1-415b-94ae-8ebb36bf9df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq" Jul 17 22:32:58.546065 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:58.546045 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d178eef0-60d1-415b-94ae-8ebb36bf9df5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq\" (UID: \"d178eef0-60d1-415b-94ae-8ebb36bf9df5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq" Jul 17 22:32:58.651007 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:58.650939 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq" Jul 17 22:32:59.002553 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:59.002528 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq"] Jul 17 22:32:59.004659 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:32:59.004630 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd178eef0_60d1_415b_94ae_8ebb36bf9df5.slice/crio-19920a0a4b88563a9259618025744d3b721a42d5e696f64a151521fbe29728d2 WatchSource:0}: Error finding container 19920a0a4b88563a9259618025744d3b721a42d5e696f64a151521fbe29728d2: Status 404 returned error can't find the container with id 19920a0a4b88563a9259618025744d3b721a42d5e696f64a151521fbe29728d2 Jul 17 22:32:59.408485 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:59.408441 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq" event={"ID":"d178eef0-60d1-415b-94ae-8ebb36bf9df5","Type":"ContainerStarted","Data":"82a361a2ab61503fa2663c8baaaaded5e816db95f6829a2893c79c3ff53a6dcc"} Jul 17 22:32:59.408904 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:59.408528 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq" event={"ID":"d178eef0-60d1-415b-94ae-8ebb36bf9df5","Type":"ContainerStarted","Data":"19920a0a4b88563a9259618025744d3b721a42d5e696f64a151521fbe29728d2"} Jul 17 22:32:59.432412 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:59.432364 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq" podStartSLOduration=1.43234997 podStartE2EDuration="1.43234997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:32:59.42924936 +0000 UTC m=+2816.289472877" watchObservedRunningTime="2026-07-17 22:32:59.43234997 +0000 UTC m=+2816.292573471" Jul 17 22:32:59.694530 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:32:59.694505 2577 scope.go:117] "RemoveContainer" containerID="912083c78f0a8bd6ae2b64aa00d219eb2624ad53697b02bc324419d5937795fa" Jul 17 22:33:00.414972 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:00.414946 2577 generic.go:356] "Generic (PLEG): container finished" podID="4036010d-699e-41ac-8167-031b0dea254c" containerID="29e4010f7f87ca1bfb5e832031f2d9e961244acfd5a408153c728797aa6640a9" exitCode=6 Jul 17 22:33:00.415343 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:00.415019 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q" event={"ID":"4036010d-699e-41ac-8167-031b0dea254c","Type":"ContainerDied","Data":"29e4010f7f87ca1bfb5e832031f2d9e961244acfd5a408153c728797aa6640a9"} Jul 17 22:33:00.415343 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:00.415074 2577 scope.go:117] "RemoveContainer" containerID="912083c78f0a8bd6ae2b64aa00d219eb2624ad53697b02bc324419d5937795fa" Jul 17 22:33:00.415462 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:00.415377 2577 scope.go:117] "RemoveContainer" containerID="29e4010f7f87ca1bfb5e832031f2d9e961244acfd5a408153c728797aa6640a9" Jul 17 22:33:00.415658 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:33:00.415631 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q_odh-ai-gateway-infra(4036010d-699e-41ac-8167-031b0dea254c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q" podUID="4036010d-699e-41ac-8167-031b0dea254c" Jul 17 22:33:00.416799 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:00.416775 2577 generic.go:356] "Generic (PLEG): container finished" podID="d178eef0-60d1-415b-94ae-8ebb36bf9df5" containerID="82a361a2ab61503fa2663c8baaaaded5e816db95f6829a2893c79c3ff53a6dcc" exitCode=6 Jul 17 22:33:00.416910 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:00.416857 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq" event={"ID":"d178eef0-60d1-415b-94ae-8ebb36bf9df5","Type":"ContainerDied","Data":"82a361a2ab61503fa2663c8baaaaded5e816db95f6829a2893c79c3ff53a6dcc"} Jul 17 22:33:00.417064 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:00.417051 2577 scope.go:117] "RemoveContainer" containerID="82a361a2ab61503fa2663c8baaaaded5e816db95f6829a2893c79c3ff53a6dcc" Jul 17 22:33:01.423445 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:01.423354 2577 generic.go:356] "Generic (PLEG): container finished" podID="d178eef0-60d1-415b-94ae-8ebb36bf9df5" containerID="f2a46f729d29bfa2fba38fced2f7827ddbbefa5c4bf250843c220a9993875caa" exitCode=6 Jul 17 22:33:01.423898 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:01.423445 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq" event={"ID":"d178eef0-60d1-415b-94ae-8ebb36bf9df5","Type":"ContainerDied","Data":"f2a46f729d29bfa2fba38fced2f7827ddbbefa5c4bf250843c220a9993875caa"} Jul 17 22:33:01.423898 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:01.423532 2577 scope.go:117] "RemoveContainer" containerID="82a361a2ab61503fa2663c8baaaaded5e816db95f6829a2893c79c3ff53a6dcc" Jul 17 22:33:01.423898 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:01.423674 2577 scope.go:117] "RemoveContainer" containerID="f2a46f729d29bfa2fba38fced2f7827ddbbefa5c4bf250843c220a9993875caa" Jul 17 22:33:01.424058 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:33:01.423987 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq_odh-ai-gateway-infra(d178eef0-60d1-415b-94ae-8ebb36bf9df5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq" podUID="d178eef0-60d1-415b-94ae-8ebb36bf9df5" Jul 17 22:33:01.467254 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:01.467229 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q"] Jul 17 22:33:01.624883 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:01.624859 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q" Jul 17 22:33:01.774450 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:01.774417 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4036010d-699e-41ac-8167-031b0dea254c-maas-api-service-ca\") pod \"4036010d-699e-41ac-8167-031b0dea254c\" (UID: \"4036010d-699e-41ac-8167-031b0dea254c\") " Jul 17 22:33:01.774814 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:01.774791 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4036010d-699e-41ac-8167-031b0dea254c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4036010d-699e-41ac-8167-031b0dea254c" (UID: "4036010d-699e-41ac-8167-031b0dea254c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:33:01.875950 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:01.875921 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4036010d-699e-41ac-8167-031b0dea254c-maas-api-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:33:02.432455 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:02.432425 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q" Jul 17 22:33:02.432959 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:02.432449 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q" event={"ID":"4036010d-699e-41ac-8167-031b0dea254c","Type":"ContainerDied","Data":"1c815afba30bb2799e22eff8b816829270e8f56a6a7d7837b5988d857a8aa2c3"} Jul 17 22:33:02.432959 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:02.432558 2577 scope.go:117] "RemoveContainer" containerID="29e4010f7f87ca1bfb5e832031f2d9e961244acfd5a408153c728797aa6640a9" Jul 17 22:33:02.434514 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:02.434471 2577 scope.go:117] "RemoveContainer" containerID="f2a46f729d29bfa2fba38fced2f7827ddbbefa5c4bf250843c220a9993875caa" Jul 17 22:33:02.434724 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:33:02.434706 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq_odh-ai-gateway-infra(d178eef0-60d1-415b-94ae-8ebb36bf9df5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq" podUID="d178eef0-60d1-415b-94ae-8ebb36bf9df5" Jul 17 22:33:02.498959 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:02.498925 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q"] Jul 17 22:33:02.513430 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:02.513396 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-8mt7q"] Jul 17 22:33:03.702234 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:03.702197 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4036010d-699e-41ac-8167-031b0dea254c" path="/var/lib/kubelet/pods/4036010d-699e-41ac-8167-031b0dea254c/volumes" Jul 17 22:33:17.694059 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:17.694029 2577 scope.go:117] "RemoveContainer" containerID="f2a46f729d29bfa2fba38fced2f7827ddbbefa5c4bf250843c220a9993875caa" Jul 17 22:33:18.521828 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:18.521793 2577 generic.go:356] "Generic (PLEG): container finished" podID="d178eef0-60d1-415b-94ae-8ebb36bf9df5" containerID="8f598a73ab3503456364b1b2f54fba2af0f77c7ff5ef8552f009ab50bb8436ad" exitCode=6 Jul 17 22:33:18.522000 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:18.521828 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq" event={"ID":"d178eef0-60d1-415b-94ae-8ebb36bf9df5","Type":"ContainerDied","Data":"8f598a73ab3503456364b1b2f54fba2af0f77c7ff5ef8552f009ab50bb8436ad"} Jul 17 22:33:18.522000 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:18.521894 2577 scope.go:117] "RemoveContainer" containerID="f2a46f729d29bfa2fba38fced2f7827ddbbefa5c4bf250843c220a9993875caa" Jul 17 22:33:18.522209 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:18.522192 2577 scope.go:117] "RemoveContainer" containerID="8f598a73ab3503456364b1b2f54fba2af0f77c7ff5ef8552f009ab50bb8436ad" Jul 17 22:33:18.522448 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:33:18.522424 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq_odh-ai-gateway-infra(d178eef0-60d1-415b-94ae-8ebb36bf9df5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq" podUID="d178eef0-60d1-415b-94ae-8ebb36bf9df5" Jul 17 22:33:19.558437 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:19.558404 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq"] Jul 17 22:33:19.698477 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:19.698435 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq" Jul 17 22:33:19.746880 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:19.746855 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d178eef0-60d1-415b-94ae-8ebb36bf9df5-maas-api-service-ca\") pod \"d178eef0-60d1-415b-94ae-8ebb36bf9df5\" (UID: \"d178eef0-60d1-415b-94ae-8ebb36bf9df5\") " Jul 17 22:33:19.747260 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:19.747237 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d178eef0-60d1-415b-94ae-8ebb36bf9df5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d178eef0-60d1-415b-94ae-8ebb36bf9df5" (UID: "d178eef0-60d1-415b-94ae-8ebb36bf9df5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:33:19.747375 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:19.747357 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d178eef0-60d1-415b-94ae-8ebb36bf9df5-maas-api-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:33:20.543011 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:20.542974 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq" Jul 17 22:33:20.543011 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:20.542985 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq" event={"ID":"d178eef0-60d1-415b-94ae-8ebb36bf9df5","Type":"ContainerDied","Data":"19920a0a4b88563a9259618025744d3b721a42d5e696f64a151521fbe29728d2"} Jul 17 22:33:20.543271 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:20.543060 2577 scope.go:117] "RemoveContainer" containerID="8f598a73ab3503456364b1b2f54fba2af0f77c7ff5ef8552f009ab50bb8436ad" Jul 17 22:33:20.564884 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:20.564859 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq"] Jul 17 22:33:20.570640 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:20.570616 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7fb802f2-j2ffq"] Jul 17 22:33:21.700818 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:21.700788 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d178eef0-60d1-415b-94ae-8ebb36bf9df5" path="/var/lib/kubelet/pods/d178eef0-60d1-415b-94ae-8ebb36bf9df5/volumes" Jul 17 22:33:32.898713 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.898672 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c"] Jul 17 22:33:32.899328 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.899306 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d178eef0-60d1-415b-94ae-8ebb36bf9df5" containerName="revoke-keys" Jul 17 22:33:32.899427 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.899331 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d178eef0-60d1-415b-94ae-8ebb36bf9df5" containerName="revoke-keys" Jul 17 22:33:32.899427 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.899355 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4036010d-699e-41ac-8167-031b0dea254c" containerName="revoke-keys" Jul 17 22:33:32.899427 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.899364 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4036010d-699e-41ac-8167-031b0dea254c" containerName="revoke-keys" Jul 17 22:33:32.899427 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.899374 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4036010d-699e-41ac-8167-031b0dea254c" containerName="revoke-keys" Jul 17 22:33:32.899427 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.899384 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4036010d-699e-41ac-8167-031b0dea254c" containerName="revoke-keys" Jul 17 22:33:32.899427 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.899408 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4036010d-699e-41ac-8167-031b0dea254c" containerName="revoke-keys" Jul 17 22:33:32.899427 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.899416 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4036010d-699e-41ac-8167-031b0dea254c" containerName="revoke-keys" Jul 17 22:33:32.899828 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.899440 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d178eef0-60d1-415b-94ae-8ebb36bf9df5" containerName="revoke-keys" Jul 17 22:33:32.899828 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.899449 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d178eef0-60d1-415b-94ae-8ebb36bf9df5" containerName="revoke-keys" Jul 17 22:33:32.899828 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.899583 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d178eef0-60d1-415b-94ae-8ebb36bf9df5" containerName="revoke-keys" Jul 17 22:33:32.899828 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.899600 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4036010d-699e-41ac-8167-031b0dea254c" containerName="revoke-keys" Jul 17 22:33:32.899828 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.899608 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4036010d-699e-41ac-8167-031b0dea254c" containerName="revoke-keys" Jul 17 22:33:32.899828 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.899618 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d178eef0-60d1-415b-94ae-8ebb36bf9df5" containerName="revoke-keys" Jul 17 22:33:32.899828 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.899628 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4036010d-699e-41ac-8167-031b0dea254c" containerName="revoke-keys" Jul 17 22:33:32.904124 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.904103 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c" Jul 17 22:33:32.906924 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.906737 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5k9rd\"" Jul 17 22:33:32.909474 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.909448 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c"] Jul 17 22:33:32.971086 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:32.971056 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/341abfbc-10fd-492f-a525-9040f8425c59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c\" (UID: \"341abfbc-10fd-492f-a525-9040f8425c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c" Jul 17 22:33:33.071784 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:33.071740 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/341abfbc-10fd-492f-a525-9040f8425c59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c\" (UID: \"341abfbc-10fd-492f-a525-9040f8425c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c" Jul 17 22:33:33.072390 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:33.072369 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/341abfbc-10fd-492f-a525-9040f8425c59-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c\" (UID: \"341abfbc-10fd-492f-a525-9040f8425c59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c" Jul 17 22:33:33.217175 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:33.217142 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c" Jul 17 22:33:33.359971 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:33.359940 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c"] Jul 17 22:33:33.360934 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:33:33.360908 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod341abfbc_10fd_492f_a525_9040f8425c59.slice/crio-9a8bd931ff9ad718d7f2889e2c3eadd3cc96380a1481644c54710e98d06197bf WatchSource:0}: Error finding container 9a8bd931ff9ad718d7f2889e2c3eadd3cc96380a1481644c54710e98d06197bf: Status 404 returned error can't find the container with id 9a8bd931ff9ad718d7f2889e2c3eadd3cc96380a1481644c54710e98d06197bf Jul 17 22:33:33.614800 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:33.614765 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c" event={"ID":"341abfbc-10fd-492f-a525-9040f8425c59","Type":"ContainerStarted","Data":"9a8bd931ff9ad718d7f2889e2c3eadd3cc96380a1481644c54710e98d06197bf"} Jul 17 22:33:34.621222 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:34.621133 2577 generic.go:356] "Generic (PLEG): container finished" podID="341abfbc-10fd-492f-a525-9040f8425c59" containerID="3c52ee0ae1d08c52103cc154765e7d3ab85d60b90bf2d923d8eabd5435005e9d" exitCode=6 Jul 17 22:33:34.621222 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:34.621175 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c" event={"ID":"341abfbc-10fd-492f-a525-9040f8425c59","Type":"ContainerDied","Data":"3c52ee0ae1d08c52103cc154765e7d3ab85d60b90bf2d923d8eabd5435005e9d"} Jul 17 22:33:34.621637 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:34.621392 2577 scope.go:117] "RemoveContainer" containerID="3c52ee0ae1d08c52103cc154765e7d3ab85d60b90bf2d923d8eabd5435005e9d" Jul 17 22:33:35.628626 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:35.628538 2577 generic.go:356] "Generic (PLEG): container finished" podID="341abfbc-10fd-492f-a525-9040f8425c59" containerID="4ea880c86807149b99154ad838a07db8a0b461fa38ed41e6fe7747e6c03ece5c" exitCode=6 Jul 17 22:33:35.629071 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:35.628626 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c" event={"ID":"341abfbc-10fd-492f-a525-9040f8425c59","Type":"ContainerDied","Data":"4ea880c86807149b99154ad838a07db8a0b461fa38ed41e6fe7747e6c03ece5c"} Jul 17 22:33:35.629071 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:35.628688 2577 scope.go:117] "RemoveContainer" containerID="3c52ee0ae1d08c52103cc154765e7d3ab85d60b90bf2d923d8eabd5435005e9d" Jul 17 22:33:35.629071 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:35.628866 2577 scope.go:117] "RemoveContainer" containerID="4ea880c86807149b99154ad838a07db8a0b461fa38ed41e6fe7747e6c03ece5c" Jul 17 22:33:35.629225 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:33:35.629121 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c_odh-ai-gateway-infra(341abfbc-10fd-492f-a525-9040f8425c59)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c" podUID="341abfbc-10fd-492f-a525-9040f8425c59" Jul 17 22:33:36.636012 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:36.635985 2577 scope.go:117] "RemoveContainer" containerID="4ea880c86807149b99154ad838a07db8a0b461fa38ed41e6fe7747e6c03ece5c" Jul 17 22:33:36.636380 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:33:36.636194 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c_odh-ai-gateway-infra(341abfbc-10fd-492f-a525-9040f8425c59)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c" podUID="341abfbc-10fd-492f-a525-9040f8425c59" Jul 17 22:33:48.694738 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:48.694712 2577 scope.go:117] "RemoveContainer" containerID="4ea880c86807149b99154ad838a07db8a0b461fa38ed41e6fe7747e6c03ece5c" Jul 17 22:33:49.719142 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:49.719110 2577 generic.go:356] "Generic (PLEG): container finished" podID="341abfbc-10fd-492f-a525-9040f8425c59" containerID="3b9b3ea43a1c7975ed75b049217c67ebed3a561d028055ac631f244381b5e39b" exitCode=6 Jul 17 22:33:49.719542 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:49.719177 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c" event={"ID":"341abfbc-10fd-492f-a525-9040f8425c59","Type":"ContainerDied","Data":"3b9b3ea43a1c7975ed75b049217c67ebed3a561d028055ac631f244381b5e39b"} Jul 17 22:33:49.719542 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:49.719230 2577 scope.go:117] "RemoveContainer" containerID="4ea880c86807149b99154ad838a07db8a0b461fa38ed41e6fe7747e6c03ece5c" Jul 17 22:33:49.719542 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:49.719540 2577 scope.go:117] "RemoveContainer" containerID="3b9b3ea43a1c7975ed75b049217c67ebed3a561d028055ac631f244381b5e39b" Jul 17 22:33:49.719770 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:33:49.719750 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c_odh-ai-gateway-infra(341abfbc-10fd-492f-a525-9040f8425c59)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c" podUID="341abfbc-10fd-492f-a525-9040f8425c59" Jul 17 22:33:50.750052 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:50.750017 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c"] Jul 17 22:33:50.888164 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:50.888142 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c" Jul 17 22:33:51.051395 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:51.051363 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/341abfbc-10fd-492f-a525-9040f8425c59-maas-api-service-ca\") pod \"341abfbc-10fd-492f-a525-9040f8425c59\" (UID: \"341abfbc-10fd-492f-a525-9040f8425c59\") " Jul 17 22:33:51.051749 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:51.051725 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/341abfbc-10fd-492f-a525-9040f8425c59-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "341abfbc-10fd-492f-a525-9040f8425c59" (UID: "341abfbc-10fd-492f-a525-9040f8425c59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:33:51.152263 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:51.152232 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/341abfbc-10fd-492f-a525-9040f8425c59-maas-api-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:33:51.733369 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:51.733338 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c" event={"ID":"341abfbc-10fd-492f-a525-9040f8425c59","Type":"ContainerDied","Data":"9a8bd931ff9ad718d7f2889e2c3eadd3cc96380a1481644c54710e98d06197bf"} Jul 17 22:33:51.733527 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:51.733397 2577 scope.go:117] "RemoveContainer" containerID="3b9b3ea43a1c7975ed75b049217c67ebed3a561d028055ac631f244381b5e39b" Jul 17 22:33:51.733527 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:51.733351 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c" Jul 17 22:33:51.750270 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:51.750244 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c"] Jul 17 22:33:51.754390 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:51.754364 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ded85f30-2lw5c"] Jul 17 22:33:53.701915 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:53.701885 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="341abfbc-10fd-492f-a525-9040f8425c59" path="/var/lib/kubelet/pods/341abfbc-10fd-492f-a525-9040f8425c59/volumes" Jul 17 22:33:54.026772 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.026661 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg"] Jul 17 22:33:54.026949 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.026899 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" podUID="299e1f73-1041-4eb4-8a07-4b76dfcae186" containerName="maas-api" containerID="cri-o://eaadfc9f76c34e15d12fc93fe3b9120fb05e1e32f47ee2eeb66b3b6cc774644f" gracePeriod=30 Jul 17 22:33:54.283476 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.283408 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" Jul 17 22:33:54.383877 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.383849 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/299e1f73-1041-4eb4-8a07-4b76dfcae186-maas-api-tls\") pod \"299e1f73-1041-4eb4-8a07-4b76dfcae186\" (UID: \"299e1f73-1041-4eb4-8a07-4b76dfcae186\") " Jul 17 22:33:54.384054 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.383886 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmj98\" (UniqueName: \"kubernetes.io/projected/299e1f73-1041-4eb4-8a07-4b76dfcae186-kube-api-access-fmj98\") pod \"299e1f73-1041-4eb4-8a07-4b76dfcae186\" (UID: \"299e1f73-1041-4eb4-8a07-4b76dfcae186\") " Jul 17 22:33:54.384054 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.383923 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/299e1f73-1041-4eb4-8a07-4b76dfcae186-openshift-service-ca\") pod \"299e1f73-1041-4eb4-8a07-4b76dfcae186\" (UID: \"299e1f73-1041-4eb4-8a07-4b76dfcae186\") " Jul 17 22:33:54.384320 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.384294 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/299e1f73-1041-4eb4-8a07-4b76dfcae186-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "299e1f73-1041-4eb4-8a07-4b76dfcae186" (UID: "299e1f73-1041-4eb4-8a07-4b76dfcae186"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:33:54.386174 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.386141 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/299e1f73-1041-4eb4-8a07-4b76dfcae186-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "299e1f73-1041-4eb4-8a07-4b76dfcae186" (UID: "299e1f73-1041-4eb4-8a07-4b76dfcae186"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:33:54.386279 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.386186 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/299e1f73-1041-4eb4-8a07-4b76dfcae186-kube-api-access-fmj98" (OuterVolumeSpecName: "kube-api-access-fmj98") pod "299e1f73-1041-4eb4-8a07-4b76dfcae186" (UID: "299e1f73-1041-4eb4-8a07-4b76dfcae186"). InnerVolumeSpecName "kube-api-access-fmj98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:33:54.484733 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.484700 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/299e1f73-1041-4eb4-8a07-4b76dfcae186-maas-api-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:33:54.484733 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.484730 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmj98\" (UniqueName: \"kubernetes.io/projected/299e1f73-1041-4eb4-8a07-4b76dfcae186-kube-api-access-fmj98\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:33:54.484926 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.484743 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/299e1f73-1041-4eb4-8a07-4b76dfcae186-openshift-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:33:54.751351 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.751313 2577 generic.go:356] "Generic (PLEG): container finished" podID="299e1f73-1041-4eb4-8a07-4b76dfcae186" containerID="eaadfc9f76c34e15d12fc93fe3b9120fb05e1e32f47ee2eeb66b3b6cc774644f" exitCode=0 Jul 17 22:33:54.751882 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.751361 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" Jul 17 22:33:54.751882 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.751406 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" event={"ID":"299e1f73-1041-4eb4-8a07-4b76dfcae186","Type":"ContainerDied","Data":"eaadfc9f76c34e15d12fc93fe3b9120fb05e1e32f47ee2eeb66b3b6cc774644f"} Jul 17 22:33:54.751882 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.751470 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg" event={"ID":"299e1f73-1041-4eb4-8a07-4b76dfcae186","Type":"ContainerDied","Data":"31f222902b79dfda647ad6210c1ac19092cda05d93ef12dde9d3b2a869529bbe"} Jul 17 22:33:54.751882 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.751523 2577 scope.go:117] "RemoveContainer" containerID="eaadfc9f76c34e15d12fc93fe3b9120fb05e1e32f47ee2eeb66b3b6cc774644f" Jul 17 22:33:54.761298 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.761279 2577 scope.go:117] "RemoveContainer" containerID="eaadfc9f76c34e15d12fc93fe3b9120fb05e1e32f47ee2eeb66b3b6cc774644f" Jul 17 22:33:54.761562 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:33:54.761544 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaadfc9f76c34e15d12fc93fe3b9120fb05e1e32f47ee2eeb66b3b6cc774644f\": container with ID starting with eaadfc9f76c34e15d12fc93fe3b9120fb05e1e32f47ee2eeb66b3b6cc774644f not found: ID does not exist" containerID="eaadfc9f76c34e15d12fc93fe3b9120fb05e1e32f47ee2eeb66b3b6cc774644f" Jul 17 22:33:54.761640 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.761575 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaadfc9f76c34e15d12fc93fe3b9120fb05e1e32f47ee2eeb66b3b6cc774644f"} err="failed to get container status \"eaadfc9f76c34e15d12fc93fe3b9120fb05e1e32f47ee2eeb66b3b6cc774644f\": rpc error: code = NotFound desc = could not find container \"eaadfc9f76c34e15d12fc93fe3b9120fb05e1e32f47ee2eeb66b3b6cc774644f\": container with ID starting with eaadfc9f76c34e15d12fc93fe3b9120fb05e1e32f47ee2eeb66b3b6cc774644f not found: ID does not exist" Jul 17 22:33:54.773340 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.773318 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg"] Jul 17 22:33:54.782057 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:54.782036 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-11aecf-6855bc745f-v4nqg"] Jul 17 22:33:55.700436 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:33:55.700401 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="299e1f73-1041-4eb4-8a07-4b76dfcae186" path="/var/lib/kubelet/pods/299e1f73-1041-4eb4-8a07-4b76dfcae186/volumes" Jul 17 22:34:20.170346 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.170311 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb"] Jul 17 22:34:20.171466 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.170580 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" podUID="cc65781e-8218-4e7a-aa97-f2a35cd84dda" containerName="maas-api" containerID="cri-o://8a6a401713a1ab7e6eaa3b00c36f1d5aa4fc3dc9c693656300e6cffc8d91ee99" gracePeriod=30 Jul 17 22:34:20.428581 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.428525 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" Jul 17 22:34:20.537972 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.537942 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5j4x\" (UniqueName: \"kubernetes.io/projected/cc65781e-8218-4e7a-aa97-f2a35cd84dda-kube-api-access-r5j4x\") pod \"cc65781e-8218-4e7a-aa97-f2a35cd84dda\" (UID: \"cc65781e-8218-4e7a-aa97-f2a35cd84dda\") " Jul 17 22:34:20.538133 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.538034 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc65781e-8218-4e7a-aa97-f2a35cd84dda-openshift-service-ca\") pod \"cc65781e-8218-4e7a-aa97-f2a35cd84dda\" (UID: \"cc65781e-8218-4e7a-aa97-f2a35cd84dda\") " Jul 17 22:34:20.538133 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.538063 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc65781e-8218-4e7a-aa97-f2a35cd84dda-maas-api-tls\") pod \"cc65781e-8218-4e7a-aa97-f2a35cd84dda\" (UID: \"cc65781e-8218-4e7a-aa97-f2a35cd84dda\") " Jul 17 22:34:20.538380 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.538354 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc65781e-8218-4e7a-aa97-f2a35cd84dda-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cc65781e-8218-4e7a-aa97-f2a35cd84dda" (UID: "cc65781e-8218-4e7a-aa97-f2a35cd84dda"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:34:20.540114 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.540090 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc65781e-8218-4e7a-aa97-f2a35cd84dda-kube-api-access-r5j4x" (OuterVolumeSpecName: "kube-api-access-r5j4x") pod "cc65781e-8218-4e7a-aa97-f2a35cd84dda" (UID: "cc65781e-8218-4e7a-aa97-f2a35cd84dda"). InnerVolumeSpecName "kube-api-access-r5j4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:34:20.540270 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.540255 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc65781e-8218-4e7a-aa97-f2a35cd84dda-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cc65781e-8218-4e7a-aa97-f2a35cd84dda" (UID: "cc65781e-8218-4e7a-aa97-f2a35cd84dda"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:34:20.638972 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.638939 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5j4x\" (UniqueName: \"kubernetes.io/projected/cc65781e-8218-4e7a-aa97-f2a35cd84dda-kube-api-access-r5j4x\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:34:20.638972 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.638966 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc65781e-8218-4e7a-aa97-f2a35cd84dda-openshift-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:34:20.638972 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.638976 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc65781e-8218-4e7a-aa97-f2a35cd84dda-maas-api-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:34:20.890350 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.890320 2577 generic.go:356] "Generic (PLEG): container finished" podID="cc65781e-8218-4e7a-aa97-f2a35cd84dda" containerID="8a6a401713a1ab7e6eaa3b00c36f1d5aa4fc3dc9c693656300e6cffc8d91ee99" exitCode=0 Jul 17 22:34:20.890571 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.890370 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" Jul 17 22:34:20.890571 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.890383 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" event={"ID":"cc65781e-8218-4e7a-aa97-f2a35cd84dda","Type":"ContainerDied","Data":"8a6a401713a1ab7e6eaa3b00c36f1d5aa4fc3dc9c693656300e6cffc8d91ee99"} Jul 17 22:34:20.890571 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.890444 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb" event={"ID":"cc65781e-8218-4e7a-aa97-f2a35cd84dda","Type":"ContainerDied","Data":"00cae66a3c23bb891b6b8e2aa6ab1f357e51934a2b039dafdfee23b3e39ea2ca"} Jul 17 22:34:20.890571 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.890473 2577 scope.go:117] "RemoveContainer" containerID="8a6a401713a1ab7e6eaa3b00c36f1d5aa4fc3dc9c693656300e6cffc8d91ee99" Jul 17 22:34:20.900887 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.900867 2577 scope.go:117] "RemoveContainer" containerID="8a6a401713a1ab7e6eaa3b00c36f1d5aa4fc3dc9c693656300e6cffc8d91ee99" Jul 17 22:34:20.901142 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:34:20.901119 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a6a401713a1ab7e6eaa3b00c36f1d5aa4fc3dc9c693656300e6cffc8d91ee99\": container with ID starting with 8a6a401713a1ab7e6eaa3b00c36f1d5aa4fc3dc9c693656300e6cffc8d91ee99 not found: ID does not exist" containerID="8a6a401713a1ab7e6eaa3b00c36f1d5aa4fc3dc9c693656300e6cffc8d91ee99" Jul 17 22:34:20.901292 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.901152 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a6a401713a1ab7e6eaa3b00c36f1d5aa4fc3dc9c693656300e6cffc8d91ee99"} err="failed to get container status \"8a6a401713a1ab7e6eaa3b00c36f1d5aa4fc3dc9c693656300e6cffc8d91ee99\": rpc error: code = NotFound desc = could not find container \"8a6a401713a1ab7e6eaa3b00c36f1d5aa4fc3dc9c693656300e6cffc8d91ee99\": container with ID starting with 8a6a401713a1ab7e6eaa3b00c36f1d5aa4fc3dc9c693656300e6cffc8d91ee99 not found: ID does not exist" Jul 17 22:34:20.910892 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.910867 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb"] Jul 17 22:34:20.912916 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:20.912896 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4ccee5-5cb5746686-995hb"] Jul 17 22:34:21.702371 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:21.702339 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc65781e-8218-4e7a-aa97-f2a35cd84dda" path="/var/lib/kubelet/pods/cc65781e-8218-4e7a-aa97-f2a35cd84dda/volumes" Jul 17 22:34:40.059386 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.059351 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g"] Jul 17 22:34:40.059800 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.059785 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="341abfbc-10fd-492f-a525-9040f8425c59" containerName="revoke-keys" Jul 17 22:34:40.059800 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.059798 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="341abfbc-10fd-492f-a525-9040f8425c59" containerName="revoke-keys" Jul 17 22:34:40.059880 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.059809 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc65781e-8218-4e7a-aa97-f2a35cd84dda" containerName="maas-api" Jul 17 22:34:40.059880 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.059817 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc65781e-8218-4e7a-aa97-f2a35cd84dda" containerName="maas-api" Jul 17 22:34:40.059880 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.059826 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="299e1f73-1041-4eb4-8a07-4b76dfcae186" containerName="maas-api" Jul 17 22:34:40.059880 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.059832 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="299e1f73-1041-4eb4-8a07-4b76dfcae186" containerName="maas-api" Jul 17 22:34:40.059880 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.059839 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="341abfbc-10fd-492f-a525-9040f8425c59" containerName="revoke-keys" Jul 17 22:34:40.059880 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.059844 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="341abfbc-10fd-492f-a525-9040f8425c59" containerName="revoke-keys" Jul 17 22:34:40.059880 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.059855 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d178eef0-60d1-415b-94ae-8ebb36bf9df5" containerName="revoke-keys" Jul 17 22:34:40.059880 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.059860 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d178eef0-60d1-415b-94ae-8ebb36bf9df5" containerName="revoke-keys" Jul 17 22:34:40.060139 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.059930 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc65781e-8218-4e7a-aa97-f2a35cd84dda" containerName="maas-api" Jul 17 22:34:40.060139 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.059937 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="299e1f73-1041-4eb4-8a07-4b76dfcae186" containerName="maas-api" Jul 17 22:34:40.060139 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.059943 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="341abfbc-10fd-492f-a525-9040f8425c59" containerName="revoke-keys" Jul 17 22:34:40.060139 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.059948 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="341abfbc-10fd-492f-a525-9040f8425c59" containerName="revoke-keys" Jul 17 22:34:40.060139 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.059955 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d178eef0-60d1-415b-94ae-8ebb36bf9df5" containerName="revoke-keys" Jul 17 22:34:40.063108 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.063091 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" Jul 17 22:34:40.068568 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.068545 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5abe32-serving-cert\"" Jul 17 22:34:40.080584 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.080557 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g"] Jul 17 22:34:40.117369 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.117339 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff61d586-bc5f-4336-a9f4-49b1562da498-openshift-service-ca\") pod \"maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g\" (UID: \"ff61d586-bc5f-4336-a9f4-49b1562da498\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" Jul 17 22:34:40.117592 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.117434 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxb7v\" (UniqueName: \"kubernetes.io/projected/ff61d586-bc5f-4336-a9f4-49b1562da498-kube-api-access-cxb7v\") pod \"maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g\" (UID: \"ff61d586-bc5f-4336-a9f4-49b1562da498\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" Jul 17 22:34:40.117592 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.117482 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff61d586-bc5f-4336-a9f4-49b1562da498-maas-api-tls\") pod \"maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g\" (UID: \"ff61d586-bc5f-4336-a9f4-49b1562da498\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" Jul 17 22:34:40.219023 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.218982 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff61d586-bc5f-4336-a9f4-49b1562da498-maas-api-tls\") pod \"maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g\" (UID: \"ff61d586-bc5f-4336-a9f4-49b1562da498\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" Jul 17 22:34:40.219210 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.219051 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff61d586-bc5f-4336-a9f4-49b1562da498-openshift-service-ca\") pod \"maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g\" (UID: \"ff61d586-bc5f-4336-a9f4-49b1562da498\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" Jul 17 22:34:40.219210 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.219101 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxb7v\" (UniqueName: \"kubernetes.io/projected/ff61d586-bc5f-4336-a9f4-49b1562da498-kube-api-access-cxb7v\") pod \"maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g\" (UID: \"ff61d586-bc5f-4336-a9f4-49b1562da498\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" Jul 17 22:34:40.219210 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:34:40.219137 2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-serving-cert: secret "maas-api-e2e-api-a-5abe32-serving-cert" not found Jul 17 22:34:40.219210 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:34:40.219198 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff61d586-bc5f-4336-a9f4-49b1562da498-maas-api-tls podName:ff61d586-bc5f-4336-a9f4-49b1562da498 nodeName:}" failed. No retries permitted until 2026-07-17 22:34:40.719181098 +0000 UTC m=+2917.579404582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff61d586-bc5f-4336-a9f4-49b1562da498-maas-api-tls") pod "maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" (UID: "ff61d586-bc5f-4336-a9f4-49b1562da498") : secret "maas-api-e2e-api-a-5abe32-serving-cert" not found Jul 17 22:34:40.219775 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.219757 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff61d586-bc5f-4336-a9f4-49b1562da498-openshift-service-ca\") pod \"maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g\" (UID: \"ff61d586-bc5f-4336-a9f4-49b1562da498\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" Jul 17 22:34:40.238093 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.238058 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxb7v\" (UniqueName: \"kubernetes.io/projected/ff61d586-bc5f-4336-a9f4-49b1562da498-kube-api-access-cxb7v\") pod \"maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g\" (UID: \"ff61d586-bc5f-4336-a9f4-49b1562da498\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" Jul 17 22:34:40.723253 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.723204 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff61d586-bc5f-4336-a9f4-49b1562da498-maas-api-tls\") pod \"maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g\" (UID: \"ff61d586-bc5f-4336-a9f4-49b1562da498\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" Jul 17 22:34:40.725852 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.725831 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff61d586-bc5f-4336-a9f4-49b1562da498-maas-api-tls\") pod \"maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g\" (UID: \"ff61d586-bc5f-4336-a9f4-49b1562da498\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" Jul 17 22:34:40.973592 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:40.973508 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" Jul 17 22:34:41.119060 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:41.119032 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g"] Jul 17 22:34:41.120633 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:34:41.120608 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff61d586_bc5f_4336_a9f4_49b1562da498.slice/crio-74630c9b4f76519cedbbd52fa66bd7d15ae68b24a2ae6cf44353004a59fd9ec8 WatchSource:0}: Error finding container 74630c9b4f76519cedbbd52fa66bd7d15ae68b24a2ae6cf44353004a59fd9ec8: Status 404 returned error can't find the container with id 74630c9b4f76519cedbbd52fa66bd7d15ae68b24a2ae6cf44353004a59fd9ec8 Jul 17 22:34:41.122023 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:41.122004 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 22:34:42.006326 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:42.006290 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" event={"ID":"ff61d586-bc5f-4336-a9f4-49b1562da498","Type":"ContainerStarted","Data":"c4007047232577861d424d4c48ed4b6bd793d3bed5df79de1e14a9d0bb2f64a9"} Jul 17 22:34:42.006540 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:42.006344 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" event={"ID":"ff61d586-bc5f-4336-a9f4-49b1562da498","Type":"ContainerStarted","Data":"74630c9b4f76519cedbbd52fa66bd7d15ae68b24a2ae6cf44353004a59fd9ec8"} Jul 17 22:34:42.006540 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:42.006375 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" Jul 17 22:34:42.025600 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:42.025559 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" podStartSLOduration=1.647610859 podStartE2EDuration="2.025543246s" totalImagesPullingTime="377.932387ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:34:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:34:42.023268382 +0000 UTC m=+2918.883491907" watchObservedRunningTime="2026-07-17 22:34:42.025543246 +0000 UTC m=+2918.885766747" Jul 17 22:34:48.018286 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:34:48.018259 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" Jul 17 22:35:11.356402 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:11.356367 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g"] Jul 17 22:35:11.356986 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:11.356656 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" podUID="ff61d586-bc5f-4336-a9f4-49b1562da498" containerName="maas-api" containerID="cri-o://c4007047232577861d424d4c48ed4b6bd793d3bed5df79de1e14a9d0bb2f64a9" gracePeriod=30 Jul 17 22:35:11.609210 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:11.609152 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" Jul 17 22:35:11.711027 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:11.710995 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff61d586-bc5f-4336-a9f4-49b1562da498-openshift-service-ca\") pod \"ff61d586-bc5f-4336-a9f4-49b1562da498\" (UID: \"ff61d586-bc5f-4336-a9f4-49b1562da498\") " Jul 17 22:35:11.711183 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:11.711047 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxb7v\" (UniqueName: \"kubernetes.io/projected/ff61d586-bc5f-4336-a9f4-49b1562da498-kube-api-access-cxb7v\") pod \"ff61d586-bc5f-4336-a9f4-49b1562da498\" (UID: \"ff61d586-bc5f-4336-a9f4-49b1562da498\") " Jul 17 22:35:11.711183 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:11.711100 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff61d586-bc5f-4336-a9f4-49b1562da498-maas-api-tls\") pod \"ff61d586-bc5f-4336-a9f4-49b1562da498\" (UID: \"ff61d586-bc5f-4336-a9f4-49b1562da498\") " Jul 17 22:35:11.711385 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:11.711355 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff61d586-bc5f-4336-a9f4-49b1562da498-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ff61d586-bc5f-4336-a9f4-49b1562da498" (UID: "ff61d586-bc5f-4336-a9f4-49b1562da498"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 17 22:35:11.713304 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:11.713273 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff61d586-bc5f-4336-a9f4-49b1562da498-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ff61d586-bc5f-4336-a9f4-49b1562da498" (UID: "ff61d586-bc5f-4336-a9f4-49b1562da498"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 17 22:35:11.713436 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:11.713311 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff61d586-bc5f-4336-a9f4-49b1562da498-kube-api-access-cxb7v" (OuterVolumeSpecName: "kube-api-access-cxb7v") pod "ff61d586-bc5f-4336-a9f4-49b1562da498" (UID: "ff61d586-bc5f-4336-a9f4-49b1562da498"). InnerVolumeSpecName "kube-api-access-cxb7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:35:11.812213 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:11.812182 2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff61d586-bc5f-4336-a9f4-49b1562da498-openshift-service-ca\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:35:11.812213 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:11.812208 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxb7v\" (UniqueName: \"kubernetes.io/projected/ff61d586-bc5f-4336-a9f4-49b1562da498-kube-api-access-cxb7v\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:35:11.812213 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:11.812219 2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff61d586-bc5f-4336-a9f4-49b1562da498-maas-api-tls\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:35:12.175616 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:12.175569 2577 generic.go:356] "Generic (PLEG): container finished" podID="ff61d586-bc5f-4336-a9f4-49b1562da498" containerID="c4007047232577861d424d4c48ed4b6bd793d3bed5df79de1e14a9d0bb2f64a9" exitCode=0 Jul 17 22:35:12.175810 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:12.175628 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" Jul 17 22:35:12.175810 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:12.175652 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" event={"ID":"ff61d586-bc5f-4336-a9f4-49b1562da498","Type":"ContainerDied","Data":"c4007047232577861d424d4c48ed4b6bd793d3bed5df79de1e14a9d0bb2f64a9"} Jul 17 22:35:12.175810 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:12.175739 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g" event={"ID":"ff61d586-bc5f-4336-a9f4-49b1562da498","Type":"ContainerDied","Data":"74630c9b4f76519cedbbd52fa66bd7d15ae68b24a2ae6cf44353004a59fd9ec8"} Jul 17 22:35:12.175810 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:12.175768 2577 scope.go:117] "RemoveContainer" containerID="c4007047232577861d424d4c48ed4b6bd793d3bed5df79de1e14a9d0bb2f64a9" Jul 17 22:35:12.185607 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:12.185589 2577 scope.go:117] "RemoveContainer" containerID="c4007047232577861d424d4c48ed4b6bd793d3bed5df79de1e14a9d0bb2f64a9" Jul 17 22:35:12.185858 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:35:12.185839 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4007047232577861d424d4c48ed4b6bd793d3bed5df79de1e14a9d0bb2f64a9\": container with ID starting with c4007047232577861d424d4c48ed4b6bd793d3bed5df79de1e14a9d0bb2f64a9 not found: ID does not exist" containerID="c4007047232577861d424d4c48ed4b6bd793d3bed5df79de1e14a9d0bb2f64a9" Jul 17 22:35:12.185941 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:12.185866 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4007047232577861d424d4c48ed4b6bd793d3bed5df79de1e14a9d0bb2f64a9"} err="failed to get container status \"c4007047232577861d424d4c48ed4b6bd793d3bed5df79de1e14a9d0bb2f64a9\": rpc error: code = NotFound desc = could not find container \"c4007047232577861d424d4c48ed4b6bd793d3bed5df79de1e14a9d0bb2f64a9\": container with ID starting with c4007047232577861d424d4c48ed4b6bd793d3bed5df79de1e14a9d0bb2f64a9 not found: ID does not exist" Jul 17 22:35:12.197364 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:12.197338 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g"] Jul 17 22:35:12.200971 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:12.200947 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5abe32-68b888d46f-qqn9g"] Jul 17 22:35:13.702207 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:35:13.702170 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff61d586-bc5f-4336-a9f4-49b1562da498" path="/var/lib/kubelet/pods/ff61d586-bc5f-4336-a9f4-49b1562da498/volumes" Jul 17 22:37:07.127342 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:07.127310 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nl27r"] Jul 17 22:37:07.129723 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:07.127795 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="341abfbc-10fd-492f-a525-9040f8425c59" containerName="revoke-keys" Jul 17 22:37:07.129723 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:07.127807 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="341abfbc-10fd-492f-a525-9040f8425c59" containerName="revoke-keys" Jul 17 22:37:07.129723 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:07.127817 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff61d586-bc5f-4336-a9f4-49b1562da498" containerName="maas-api" Jul 17 22:37:07.129723 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:07.127822 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff61d586-bc5f-4336-a9f4-49b1562da498" containerName="maas-api" Jul 17 22:37:07.129723 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:07.127930 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff61d586-bc5f-4336-a9f4-49b1562da498" containerName="maas-api" Jul 17 22:37:07.129723 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:07.127943 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="341abfbc-10fd-492f-a525-9040f8425c59" containerName="revoke-keys" Jul 17 22:37:07.130991 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:07.130973 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nl27r" Jul 17 22:37:07.141344 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:07.141310 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nl27r"] Jul 17 22:37:07.286856 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:07.286822 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6bnv\" (UniqueName: \"kubernetes.io/projected/ed884553-bada-4d4b-9cf6-5f53177e14fc-kube-api-access-l6bnv\") pod \"kuadrant-operator-catalog-nl27r\" (UID: \"ed884553-bada-4d4b-9cf6-5f53177e14fc\") " pod="kuadrant-system/kuadrant-operator-catalog-nl27r" Jul 17 22:37:07.387646 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:07.387560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6bnv\" (UniqueName: \"kubernetes.io/projected/ed884553-bada-4d4b-9cf6-5f53177e14fc-kube-api-access-l6bnv\") pod \"kuadrant-operator-catalog-nl27r\" (UID: \"ed884553-bada-4d4b-9cf6-5f53177e14fc\") " pod="kuadrant-system/kuadrant-operator-catalog-nl27r" Jul 17 22:37:07.397331 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:07.397301 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6bnv\" (UniqueName: \"kubernetes.io/projected/ed884553-bada-4d4b-9cf6-5f53177e14fc-kube-api-access-l6bnv\") pod \"kuadrant-operator-catalog-nl27r\" (UID: \"ed884553-bada-4d4b-9cf6-5f53177e14fc\") " pod="kuadrant-system/kuadrant-operator-catalog-nl27r" Jul 17 22:37:07.442422 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:07.442388 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nl27r" Jul 17 22:37:07.577898 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:07.577835 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nl27r"] Jul 17 22:37:07.580807 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:37:07.580773 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded884553_bada_4d4b_9cf6_5f53177e14fc.slice/crio-77cd517c31c4e41426d02064302877b947623ed201cc3bf13d785caff4e71145 WatchSource:0}: Error finding container 77cd517c31c4e41426d02064302877b947623ed201cc3bf13d785caff4e71145: Status 404 returned error can't find the container with id 77cd517c31c4e41426d02064302877b947623ed201cc3bf13d785caff4e71145 Jul 17 22:37:07.772777 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:07.772744 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nl27r" event={"ID":"ed884553-bada-4d4b-9cf6-5f53177e14fc","Type":"ContainerStarted","Data":"77cd517c31c4e41426d02064302877b947623ed201cc3bf13d785caff4e71145"} Jul 17 22:37:08.780104 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:08.780064 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nl27r" event={"ID":"ed884553-bada-4d4b-9cf6-5f53177e14fc","Type":"ContainerStarted","Data":"7cdfed1f4b4b5866ea1c309f768bad75dcd6647c663736b59e166274e1c6487d"} Jul 17 22:37:08.800260 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:08.800209 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-nl27r" podStartSLOduration=1.43478767 podStartE2EDuration="1.800192492s" totalImagesPullingTime="365.404822ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:37:08.797258463 +0000 UTC m=+3065.657481965" watchObservedRunningTime="2026-07-17 22:37:08.800192492 +0000 UTC m=+3065.660415993" Jul 17 22:37:17.443107 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:17.443071 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-nl27r" Jul 17 22:37:17.443613 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:17.443132 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-nl27r" Jul 17 22:37:17.466005 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:17.465980 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-nl27r" Jul 17 22:37:17.853163 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:17.853090 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-nl27r" Jul 17 22:37:17.891256 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:17.891225 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nl27r"] Jul 17 22:37:19.840309 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:19.840268 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-nl27r" podUID="ed884553-bada-4d4b-9cf6-5f53177e14fc" containerName="registry-server" containerID="cri-o://7cdfed1f4b4b5866ea1c309f768bad75dcd6647c663736b59e166274e1c6487d" gracePeriod=2 Jul 17 22:37:20.103924 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:20.103856 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nl27r" Jul 17 22:37:20.108797 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:20.108774 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6bnv\" (UniqueName: \"kubernetes.io/projected/ed884553-bada-4d4b-9cf6-5f53177e14fc-kube-api-access-l6bnv\") pod \"ed884553-bada-4d4b-9cf6-5f53177e14fc\" (UID: \"ed884553-bada-4d4b-9cf6-5f53177e14fc\") " Jul 17 22:37:20.111393 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:20.111360 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed884553-bada-4d4b-9cf6-5f53177e14fc-kube-api-access-l6bnv" (OuterVolumeSpecName: "kube-api-access-l6bnv") pod "ed884553-bada-4d4b-9cf6-5f53177e14fc" (UID: "ed884553-bada-4d4b-9cf6-5f53177e14fc"). InnerVolumeSpecName "kube-api-access-l6bnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:37:20.210359 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:20.210320 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6bnv\" (UniqueName: \"kubernetes.io/projected/ed884553-bada-4d4b-9cf6-5f53177e14fc-kube-api-access-l6bnv\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:37:20.847120 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:20.847083 2577 generic.go:356] "Generic (PLEG): container finished" podID="ed884553-bada-4d4b-9cf6-5f53177e14fc" containerID="7cdfed1f4b4b5866ea1c309f768bad75dcd6647c663736b59e166274e1c6487d" exitCode=0 Jul 17 22:37:20.847549 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:20.847128 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-nl27r" Jul 17 22:37:20.847549 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:20.847172 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nl27r" event={"ID":"ed884553-bada-4d4b-9cf6-5f53177e14fc","Type":"ContainerDied","Data":"7cdfed1f4b4b5866ea1c309f768bad75dcd6647c663736b59e166274e1c6487d"} Jul 17 22:37:20.847549 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:20.847233 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-nl27r" event={"ID":"ed884553-bada-4d4b-9cf6-5f53177e14fc","Type":"ContainerDied","Data":"77cd517c31c4e41426d02064302877b947623ed201cc3bf13d785caff4e71145"} Jul 17 22:37:20.847549 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:20.847258 2577 scope.go:117] "RemoveContainer" containerID="7cdfed1f4b4b5866ea1c309f768bad75dcd6647c663736b59e166274e1c6487d" Jul 17 22:37:20.857047 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:20.857032 2577 scope.go:117] "RemoveContainer" containerID="7cdfed1f4b4b5866ea1c309f768bad75dcd6647c663736b59e166274e1c6487d" Jul 17 22:37:20.857307 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:37:20.857288 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cdfed1f4b4b5866ea1c309f768bad75dcd6647c663736b59e166274e1c6487d\": container with ID starting with 7cdfed1f4b4b5866ea1c309f768bad75dcd6647c663736b59e166274e1c6487d not found: ID does not exist" containerID="7cdfed1f4b4b5866ea1c309f768bad75dcd6647c663736b59e166274e1c6487d" Jul 17 22:37:20.857357 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:20.857316 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cdfed1f4b4b5866ea1c309f768bad75dcd6647c663736b59e166274e1c6487d"} err="failed to get container status \"7cdfed1f4b4b5866ea1c309f768bad75dcd6647c663736b59e166274e1c6487d\": rpc error: code = NotFound desc = could not find container \"7cdfed1f4b4b5866ea1c309f768bad75dcd6647c663736b59e166274e1c6487d\": container with ID starting with 7cdfed1f4b4b5866ea1c309f768bad75dcd6647c663736b59e166274e1c6487d not found: ID does not exist" Jul 17 22:37:20.868546 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:20.868524 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nl27r"] Jul 17 22:37:20.871856 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:20.871839 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-nl27r"] Jul 17 22:37:21.700270 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:37:21.700240 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed884553-bada-4d4b-9cf6-5f53177e14fc" path="/var/lib/kubelet/pods/ed884553-bada-4d4b-9cf6-5f53177e14fc/volumes" Jul 17 22:41:58.621942 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:41:58.621905 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7536-139695979168432"] Jul 17 22:41:58.622395 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:41:58.622329 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed884553-bada-4d4b-9cf6-5f53177e14fc" containerName="registry-server" Jul 17 22:41:58.622395 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:41:58.622340 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed884553-bada-4d4b-9cf6-5f53177e14fc" containerName="registry-server" Jul 17 22:41:58.622474 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:41:58.622409 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed884553-bada-4d4b-9cf6-5f53177e14fc" containerName="registry-server" Jul 17 22:41:58.625502 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:41:58.625474 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979168432" Jul 17 22:41:58.627930 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:41:58.627910 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4rjw7\"" Jul 17 22:41:58.640367 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:41:58.635053 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7536-139695979168432"] Jul 17 22:41:58.718616 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:41:58.718579 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ngvb\" (UniqueName: \"kubernetes.io/projected/c4f73667-c993-42b5-ac16-508056fa227a-kube-api-access-5ngvb\") pod \"test-curl-7536-139695979168432\" (UID: \"c4f73667-c993-42b5-ac16-508056fa227a\") " pod="opendatahub/test-curl-7536-139695979168432" Jul 17 22:41:58.819879 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:41:58.819841 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ngvb\" (UniqueName: \"kubernetes.io/projected/c4f73667-c993-42b5-ac16-508056fa227a-kube-api-access-5ngvb\") pod \"test-curl-7536-139695979168432\" (UID: \"c4f73667-c993-42b5-ac16-508056fa227a\") " pod="opendatahub/test-curl-7536-139695979168432" Jul 17 22:41:58.829040 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:41:58.829012 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ngvb\" (UniqueName: \"kubernetes.io/projected/c4f73667-c993-42b5-ac16-508056fa227a-kube-api-access-5ngvb\") pod \"test-curl-7536-139695979168432\" (UID: \"c4f73667-c993-42b5-ac16-508056fa227a\") " pod="opendatahub/test-curl-7536-139695979168432" Jul 17 22:41:58.941979 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:41:58.941942 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979168432" Jul 17 22:41:59.083330 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:41:59.083310 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7536-139695979168432"] Jul 17 22:41:59.086294 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:41:59.086254 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4f73667_c993_42b5_ac16_508056fa227a.slice/crio-295b5ad95ea23ca5b0e594c9061d37c85ca060dc8601a02187d5013587cac4fd WatchSource:0}: Error finding container 295b5ad95ea23ca5b0e594c9061d37c85ca060dc8601a02187d5013587cac4fd: Status 404 returned error can't find the container with id 295b5ad95ea23ca5b0e594c9061d37c85ca060dc8601a02187d5013587cac4fd Jul 17 22:41:59.087667 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:41:59.087647 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 17 22:41:59.270810 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:41:59.270725 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7536-139695979168432" event={"ID":"c4f73667-c993-42b5-ac16-508056fa227a","Type":"ContainerStarted","Data":"295b5ad95ea23ca5b0e594c9061d37c85ca060dc8601a02187d5013587cac4fd"} Jul 17 22:42:00.277374 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:00.277336 2577 generic.go:356] "Generic (PLEG): container finished" podID="c4f73667-c993-42b5-ac16-508056fa227a" containerID="44635f089f0c4dfabf9306db66f0ac03f7011d325ed0542772276eac4e90ea62" exitCode=0 Jul 17 22:42:00.277816 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:00.277423 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7536-139695979168432" event={"ID":"c4f73667-c993-42b5-ac16-508056fa227a","Type":"ContainerDied","Data":"44635f089f0c4dfabf9306db66f0ac03f7011d325ed0542772276eac4e90ea62"} Jul 17 22:42:01.425529 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:01.425479 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979168432" Jul 17 22:42:01.464673 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:01.464643 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7536-139695979168432_c4f73667-c993-42b5-ac16-508056fa227a/test-curl-7536-139695979168432/0.log" Jul 17 22:42:01.492300 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:01.492268 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7536-139695979168432"] Jul 17 22:42:01.495371 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:01.495348 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7536-139695979168432"] Jul 17 22:42:01.549004 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:01.548930 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ngvb\" (UniqueName: \"kubernetes.io/projected/c4f73667-c993-42b5-ac16-508056fa227a-kube-api-access-5ngvb\") pod \"c4f73667-c993-42b5-ac16-508056fa227a\" (UID: \"c4f73667-c993-42b5-ac16-508056fa227a\") " Jul 17 22:42:01.551163 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:01.551140 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f73667-c993-42b5-ac16-508056fa227a-kube-api-access-5ngvb" (OuterVolumeSpecName: "kube-api-access-5ngvb") pod "c4f73667-c993-42b5-ac16-508056fa227a" (UID: "c4f73667-c993-42b5-ac16-508056fa227a"). InnerVolumeSpecName "kube-api-access-5ngvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:42:01.619441 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:01.619403 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7536-139695979029552"] Jul 17 22:42:01.619916 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:01.619897 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4f73667-c993-42b5-ac16-508056fa227a" containerName="test-curl-7536-139695979168432" Jul 17 22:42:01.620008 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:01.619918 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4f73667-c993-42b5-ac16-508056fa227a" containerName="test-curl-7536-139695979168432" Jul 17 22:42:01.620068 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:01.620021 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4f73667-c993-42b5-ac16-508056fa227a" containerName="test-curl-7536-139695979168432" Jul 17 22:42:01.623219 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:01.623200 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979029552" Jul 17 22:42:01.632160 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:01.632135 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7536-139695979029552"] Jul 17 22:42:01.650157 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:01.650126 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ngvb\" (UniqueName: \"kubernetes.io/projected/c4f73667-c993-42b5-ac16-508056fa227a-kube-api-access-5ngvb\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:42:01.701280 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:01.701246 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4f73667-c993-42b5-ac16-508056fa227a" path="/var/lib/kubelet/pods/c4f73667-c993-42b5-ac16-508056fa227a/volumes" Jul 17 22:42:01.751225 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:01.751190 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-469zd\" (UniqueName: \"kubernetes.io/projected/4e647401-42e3-4718-835c-4dc04bedca99-kube-api-access-469zd\") pod \"test-curl-7536-139695979029552\" (UID: \"4e647401-42e3-4718-835c-4dc04bedca99\") " pod="opendatahub/test-curl-7536-139695979029552" Jul 17 22:42:01.852257 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:01.852162 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-469zd\" (UniqueName: \"kubernetes.io/projected/4e647401-42e3-4718-835c-4dc04bedca99-kube-api-access-469zd\") pod \"test-curl-7536-139695979029552\" (UID: \"4e647401-42e3-4718-835c-4dc04bedca99\") " pod="opendatahub/test-curl-7536-139695979029552" Jul 17 22:42:01.860405 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:01.860377 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-469zd\" (UniqueName: \"kubernetes.io/projected/4e647401-42e3-4718-835c-4dc04bedca99-kube-api-access-469zd\") pod \"test-curl-7536-139695979029552\" (UID: \"4e647401-42e3-4718-835c-4dc04bedca99\") " pod="opendatahub/test-curl-7536-139695979029552" Jul 17 22:42:01.933798 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:01.933767 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979029552" Jul 17 22:42:02.073869 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:02.073842 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7536-139695979029552"] Jul 17 22:42:02.075561 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:42:02.075527 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e647401_42e3_4718_835c_4dc04bedca99.slice/crio-fadd96ea1562046d71abfa0cbe15c919142d44bcc44ada5f8c5267cdf9a63c13 WatchSource:0}: Error finding container fadd96ea1562046d71abfa0cbe15c919142d44bcc44ada5f8c5267cdf9a63c13: Status 404 returned error can't find the container with id fadd96ea1562046d71abfa0cbe15c919142d44bcc44ada5f8c5267cdf9a63c13 Jul 17 22:42:02.294795 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:02.294765 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979168432" Jul 17 22:42:02.294995 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:02.294805 2577 scope.go:117] "RemoveContainer" containerID="44635f089f0c4dfabf9306db66f0ac03f7011d325ed0542772276eac4e90ea62" Jul 17 22:42:02.296174 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:02.295994 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7536-139695979029552" event={"ID":"4e647401-42e3-4718-835c-4dc04bedca99","Type":"ContainerStarted","Data":"fadd96ea1562046d71abfa0cbe15c919142d44bcc44ada5f8c5267cdf9a63c13"} Jul 17 22:42:03.302805 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:03.302767 2577 generic.go:356] "Generic (PLEG): container finished" podID="4e647401-42e3-4718-835c-4dc04bedca99" containerID="df2b9cda8ec57ffa810ea9c6646e89675f214c138034531ecb744056b688b8cb" exitCode=0 Jul 17 22:42:03.303240 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:03.302852 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7536-139695979029552" event={"ID":"4e647401-42e3-4718-835c-4dc04bedca99","Type":"ContainerDied","Data":"df2b9cda8ec57ffa810ea9c6646e89675f214c138034531ecb744056b688b8cb"} Jul 17 22:42:04.447376 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:04.447347 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979029552" Jul 17 22:42:04.467369 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:04.467340 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7536-139695979029552_4e647401-42e3-4718-835c-4dc04bedca99/test-curl-7536-139695979029552/0.log" Jul 17 22:42:04.492067 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:04.492030 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7536-139695979029552"] Jul 17 22:42:04.495937 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:04.495913 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7536-139695979029552"] Jul 17 22:42:04.575994 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:04.575961 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-469zd\" (UniqueName: \"kubernetes.io/projected/4e647401-42e3-4718-835c-4dc04bedca99-kube-api-access-469zd\") pod \"4e647401-42e3-4718-835c-4dc04bedca99\" (UID: \"4e647401-42e3-4718-835c-4dc04bedca99\") " Jul 17 22:42:04.590594 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:04.590551 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e647401-42e3-4718-835c-4dc04bedca99-kube-api-access-469zd" (OuterVolumeSpecName: "kube-api-access-469zd") pod "4e647401-42e3-4718-835c-4dc04bedca99" (UID: "4e647401-42e3-4718-835c-4dc04bedca99"). InnerVolumeSpecName "kube-api-access-469zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:42:04.657327 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:04.657247 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7536-139695979214144"] Jul 17 22:42:04.657747 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:04.657731 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e647401-42e3-4718-835c-4dc04bedca99" containerName="test-curl-7536-139695979029552" Jul 17 22:42:04.657800 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:04.657748 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e647401-42e3-4718-835c-4dc04bedca99" containerName="test-curl-7536-139695979029552" Jul 17 22:42:04.657838 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:04.657827 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e647401-42e3-4718-835c-4dc04bedca99" containerName="test-curl-7536-139695979029552" Jul 17 22:42:04.661044 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:04.661020 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979214144" Jul 17 22:42:04.668260 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:04.668231 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7536-139695979214144"] Jul 17 22:42:04.676691 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:04.676666 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-469zd\" (UniqueName: \"kubernetes.io/projected/4e647401-42e3-4718-835c-4dc04bedca99-kube-api-access-469zd\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:42:04.777231 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:04.777194 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z64rv\" (UniqueName: \"kubernetes.io/projected/056b65d5-8c2d-4e12-926c-311cc880f708-kube-api-access-z64rv\") pod \"test-curl-7536-139695979214144\" (UID: \"056b65d5-8c2d-4e12-926c-311cc880f708\") " pod="opendatahub/test-curl-7536-139695979214144" Jul 17 22:42:04.878380 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:04.878345 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z64rv\" (UniqueName: \"kubernetes.io/projected/056b65d5-8c2d-4e12-926c-311cc880f708-kube-api-access-z64rv\") pod \"test-curl-7536-139695979214144\" (UID: \"056b65d5-8c2d-4e12-926c-311cc880f708\") " pod="opendatahub/test-curl-7536-139695979214144" Jul 17 22:42:04.886884 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:04.886858 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z64rv\" (UniqueName: \"kubernetes.io/projected/056b65d5-8c2d-4e12-926c-311cc880f708-kube-api-access-z64rv\") pod \"test-curl-7536-139695979214144\" (UID: \"056b65d5-8c2d-4e12-926c-311cc880f708\") " pod="opendatahub/test-curl-7536-139695979214144" Jul 17 22:42:04.971860 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:04.971832 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979214144" Jul 17 22:42:05.110303 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:05.110276 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7536-139695979214144"] Jul 17 22:42:05.112380 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:42:05.112351 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod056b65d5_8c2d_4e12_926c_311cc880f708.slice/crio-311dc33f8dcb64efee8f8487e46290a61faa8cec1873e549f782e654e16779a6 WatchSource:0}: Error finding container 311dc33f8dcb64efee8f8487e46290a61faa8cec1873e549f782e654e16779a6: Status 404 returned error can't find the container with id 311dc33f8dcb64efee8f8487e46290a61faa8cec1873e549f782e654e16779a6 Jul 17 22:42:05.316059 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:05.315976 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979029552" Jul 17 22:42:05.316059 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:05.316009 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fadd96ea1562046d71abfa0cbe15c919142d44bcc44ada5f8c5267cdf9a63c13" Jul 17 22:42:05.317272 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:05.317245 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7536-139695979214144" event={"ID":"056b65d5-8c2d-4e12-926c-311cc880f708","Type":"ContainerStarted","Data":"311dc33f8dcb64efee8f8487e46290a61faa8cec1873e549f782e654e16779a6"} Jul 17 22:42:05.326883 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:42:05.326857 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7536-139695979029552\" is forbidden: User \"system:node:ip-10-0-131-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-14.ec2.internal' and this object" podUID="4e647401-42e3-4718-835c-4dc04bedca99" pod="opendatahub/test-curl-7536-139695979029552" Jul 17 22:42:05.701403 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:05.701366 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e647401-42e3-4718-835c-4dc04bedca99" path="/var/lib/kubelet/pods/4e647401-42e3-4718-835c-4dc04bedca99/volumes" Jul 17 22:42:06.323872 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:06.323841 2577 generic.go:356] "Generic (PLEG): container finished" podID="056b65d5-8c2d-4e12-926c-311cc880f708" containerID="8eb8c35352c0cb5f0365c33eee21c7f5f30219f931369fba5c6fc3c2bb80b1e4" exitCode=0 Jul 17 22:42:06.324038 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:06.323880 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7536-139695979214144" event={"ID":"056b65d5-8c2d-4e12-926c-311cc880f708","Type":"ContainerDied","Data":"8eb8c35352c0cb5f0365c33eee21c7f5f30219f931369fba5c6fc3c2bb80b1e4"} Jul 17 22:42:07.467566 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:07.467543 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979214144" Jul 17 22:42:07.489040 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:07.489006 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7536-139695979214144_056b65d5-8c2d-4e12-926c-311cc880f708/test-curl-7536-139695979214144/0.log" Jul 17 22:42:07.514007 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:07.513977 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7536-139695979214144"] Jul 17 22:42:07.518095 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:07.518068 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7536-139695979214144"] Jul 17 22:42:07.604191 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:07.604117 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z64rv\" (UniqueName: \"kubernetes.io/projected/056b65d5-8c2d-4e12-926c-311cc880f708-kube-api-access-z64rv\") pod \"056b65d5-8c2d-4e12-926c-311cc880f708\" (UID: \"056b65d5-8c2d-4e12-926c-311cc880f708\") " Jul 17 22:42:07.606261 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:07.606238 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056b65d5-8c2d-4e12-926c-311cc880f708-kube-api-access-z64rv" (OuterVolumeSpecName: "kube-api-access-z64rv") pod "056b65d5-8c2d-4e12-926c-311cc880f708" (UID: "056b65d5-8c2d-4e12-926c-311cc880f708"). InnerVolumeSpecName "kube-api-access-z64rv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:42:07.636939 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:07.636909 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7536-139695979214144"] Jul 17 22:42:07.637413 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:07.637395 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="056b65d5-8c2d-4e12-926c-311cc880f708" containerName="test-curl-7536-139695979214144" Jul 17 22:42:07.637518 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:07.637415 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="056b65d5-8c2d-4e12-926c-311cc880f708" containerName="test-curl-7536-139695979214144" Jul 17 22:42:07.637590 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:07.637568 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="056b65d5-8c2d-4e12-926c-311cc880f708" containerName="test-curl-7536-139695979214144" Jul 17 22:42:07.640657 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:07.640638 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979214144" Jul 17 22:42:07.647407 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:07.647385 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7536-139695979214144"] Jul 17 22:42:07.701293 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:07.701239 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="056b65d5-8c2d-4e12-926c-311cc880f708" path="/var/lib/kubelet/pods/056b65d5-8c2d-4e12-926c-311cc880f708/volumes" Jul 17 22:42:07.705462 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:07.705428 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzjzw\" (UniqueName: \"kubernetes.io/projected/e35f9a8e-4eca-491c-ab89-2584bcf5c5f7-kube-api-access-xzjzw\") pod \"test-curl-7536-139695979214144\" (UID: \"e35f9a8e-4eca-491c-ab89-2584bcf5c5f7\") " pod="opendatahub/test-curl-7536-139695979214144" Jul 17 22:42:07.705675 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:07.705652 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z64rv\" (UniqueName: \"kubernetes.io/projected/056b65d5-8c2d-4e12-926c-311cc880f708-kube-api-access-z64rv\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:42:07.806097 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:07.806061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzjzw\" (UniqueName: \"kubernetes.io/projected/e35f9a8e-4eca-491c-ab89-2584bcf5c5f7-kube-api-access-xzjzw\") pod \"test-curl-7536-139695979214144\" (UID: \"e35f9a8e-4eca-491c-ab89-2584bcf5c5f7\") " pod="opendatahub/test-curl-7536-139695979214144" Jul 17 22:42:07.814102 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:07.814074 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzjzw\" (UniqueName: \"kubernetes.io/projected/e35f9a8e-4eca-491c-ab89-2584bcf5c5f7-kube-api-access-xzjzw\") pod \"test-curl-7536-139695979214144\" (UID: \"e35f9a8e-4eca-491c-ab89-2584bcf5c5f7\") " pod="opendatahub/test-curl-7536-139695979214144" Jul 17 22:42:07.950439 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:07.950401 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979214144" Jul 17 22:42:08.086015 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:08.085990 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7536-139695979214144"] Jul 17 22:42:08.089617 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:42:08.089585 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode35f9a8e_4eca_491c_ab89_2584bcf5c5f7.slice/crio-33f5b9b3265c6b4a6f8effeada5551d1bbce15f286c31a9fb5a879c730d91edb WatchSource:0}: Error finding container 33f5b9b3265c6b4a6f8effeada5551d1bbce15f286c31a9fb5a879c730d91edb: Status 404 returned error can't find the container with id 33f5b9b3265c6b4a6f8effeada5551d1bbce15f286c31a9fb5a879c730d91edb Jul 17 22:42:08.336667 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:08.336590 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7536-139695979214144" event={"ID":"e35f9a8e-4eca-491c-ab89-2584bcf5c5f7","Type":"ContainerStarted","Data":"33f5b9b3265c6b4a6f8effeada5551d1bbce15f286c31a9fb5a879c730d91edb"} Jul 17 22:42:08.337793 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:08.337774 2577 scope.go:117] "RemoveContainer" containerID="8eb8c35352c0cb5f0365c33eee21c7f5f30219f931369fba5c6fc3c2bb80b1e4" Jul 17 22:42:08.337946 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:08.337822 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979214144" Jul 17 22:42:09.344741 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:09.344709 2577 generic.go:356] "Generic (PLEG): container finished" podID="e35f9a8e-4eca-491c-ab89-2584bcf5c5f7" containerID="5df7a600e9e5b26285ceaa11a6ef9e5127ec98d60dbcb017480476446c18c679" exitCode=0 Jul 17 22:42:09.345187 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:09.344811 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7536-139695979214144" event={"ID":"e35f9a8e-4eca-491c-ab89-2584bcf5c5f7","Type":"ContainerDied","Data":"5df7a600e9e5b26285ceaa11a6ef9e5127ec98d60dbcb017480476446c18c679"} Jul 17 22:42:10.491845 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:10.491820 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979214144" Jul 17 22:42:10.515351 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:10.515325 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7536-139695979214144_e35f9a8e-4eca-491c-ab89-2584bcf5c5f7/test-curl-7536-139695979214144/0.log" Jul 17 22:42:10.545925 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:10.545895 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7536-139695979214144"] Jul 17 22:42:10.550084 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:10.550054 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7536-139695979214144"] Jul 17 22:42:10.633078 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:10.632992 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzjzw\" (UniqueName: \"kubernetes.io/projected/e35f9a8e-4eca-491c-ab89-2584bcf5c5f7-kube-api-access-xzjzw\") pod \"e35f9a8e-4eca-491c-ab89-2584bcf5c5f7\" (UID: \"e35f9a8e-4eca-491c-ab89-2584bcf5c5f7\") " Jul 17 22:42:10.635396 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:10.635367 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e35f9a8e-4eca-491c-ab89-2584bcf5c5f7-kube-api-access-xzjzw" (OuterVolumeSpecName: "kube-api-access-xzjzw") pod "e35f9a8e-4eca-491c-ab89-2584bcf5c5f7" (UID: "e35f9a8e-4eca-491c-ab89-2584bcf5c5f7"). InnerVolumeSpecName "kube-api-access-xzjzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:42:10.672801 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:10.672759 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7536-139695979215152"] Jul 17 22:42:10.673255 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:10.673236 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e35f9a8e-4eca-491c-ab89-2584bcf5c5f7" containerName="test-curl-7536-139695979214144" Jul 17 22:42:10.673355 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:10.673257 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e35f9a8e-4eca-491c-ab89-2584bcf5c5f7" containerName="test-curl-7536-139695979214144" Jul 17 22:42:10.673411 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:10.673363 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e35f9a8e-4eca-491c-ab89-2584bcf5c5f7" containerName="test-curl-7536-139695979214144" Jul 17 22:42:10.676387 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:10.676367 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979215152" Jul 17 22:42:10.683910 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:10.683883 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7536-139695979215152"] Jul 17 22:42:10.734798 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:10.734763 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td2nl\" (UniqueName: \"kubernetes.io/projected/cb919ff3-4796-4b2a-ba26-633c832a77f9-kube-api-access-td2nl\") pod \"test-curl-7536-139695979215152\" (UID: \"cb919ff3-4796-4b2a-ba26-633c832a77f9\") " pod="opendatahub/test-curl-7536-139695979215152" Jul 17 22:42:10.734971 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:10.734813 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzjzw\" (UniqueName: \"kubernetes.io/projected/e35f9a8e-4eca-491c-ab89-2584bcf5c5f7-kube-api-access-xzjzw\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:42:10.835641 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:10.835602 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td2nl\" (UniqueName: \"kubernetes.io/projected/cb919ff3-4796-4b2a-ba26-633c832a77f9-kube-api-access-td2nl\") pod \"test-curl-7536-139695979215152\" (UID: \"cb919ff3-4796-4b2a-ba26-633c832a77f9\") " pod="opendatahub/test-curl-7536-139695979215152" Jul 17 22:42:10.843799 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:10.843769 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td2nl\" (UniqueName: \"kubernetes.io/projected/cb919ff3-4796-4b2a-ba26-633c832a77f9-kube-api-access-td2nl\") pod \"test-curl-7536-139695979215152\" (UID: \"cb919ff3-4796-4b2a-ba26-633c832a77f9\") " pod="opendatahub/test-curl-7536-139695979215152" Jul 17 22:42:10.986791 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:10.986757 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979215152" Jul 17 22:42:11.123361 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:11.123334 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7536-139695979215152"] Jul 17 22:42:11.125844 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:42:11.125818 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb919ff3_4796_4b2a_ba26_633c832a77f9.slice/crio-9de4f941dbb7f4cd95e993551c336dea4a5a52ab886f797ee0389382955e64e4 WatchSource:0}: Error finding container 9de4f941dbb7f4cd95e993551c336dea4a5a52ab886f797ee0389382955e64e4: Status 404 returned error can't find the container with id 9de4f941dbb7f4cd95e993551c336dea4a5a52ab886f797ee0389382955e64e4 Jul 17 22:42:11.357718 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:11.357636 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979214144" Jul 17 22:42:11.357718 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:11.357635 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33f5b9b3265c6b4a6f8effeada5551d1bbce15f286c31a9fb5a879c730d91edb" Jul 17 22:42:11.358809 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:11.358781 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7536-139695979215152" event={"ID":"cb919ff3-4796-4b2a-ba26-633c832a77f9","Type":"ContainerStarted","Data":"9de4f941dbb7f4cd95e993551c336dea4a5a52ab886f797ee0389382955e64e4"} Jul 17 22:42:11.368277 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:42:11.368249 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7536-139695979214144\" is forbidden: User \"system:node:ip-10-0-131-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-14.ec2.internal' and this object" podUID="e35f9a8e-4eca-491c-ab89-2584bcf5c5f7" pod="opendatahub/test-curl-7536-139695979214144" Jul 17 22:42:11.702238 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:11.702206 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e35f9a8e-4eca-491c-ab89-2584bcf5c5f7" path="/var/lib/kubelet/pods/e35f9a8e-4eca-491c-ab89-2584bcf5c5f7/volumes" Jul 17 22:42:12.365553 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:12.365513 2577 generic.go:356] "Generic (PLEG): container finished" podID="cb919ff3-4796-4b2a-ba26-633c832a77f9" containerID="551357d0844c3668a6e8a8cb3658b646e465a2ac695bcd01b033fbc0836cd20a" exitCode=0 Jul 17 22:42:12.365757 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:12.365585 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7536-139695979215152" event={"ID":"cb919ff3-4796-4b2a-ba26-633c832a77f9","Type":"ContainerDied","Data":"551357d0844c3668a6e8a8cb3658b646e465a2ac695bcd01b033fbc0836cd20a"} Jul 17 22:42:13.511164 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:13.511140 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979215152" Jul 17 22:42:13.533403 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:13.533369 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7536-139695979215152_cb919ff3-4796-4b2a-ba26-633c832a77f9/test-curl-7536-139695979215152/0.log" Jul 17 22:42:13.556514 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:13.556453 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7536-139695979215152"] Jul 17 22:42:13.559122 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:13.559098 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7536-139695979215152"] Jul 17 22:42:13.664953 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:13.664860 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td2nl\" (UniqueName: \"kubernetes.io/projected/cb919ff3-4796-4b2a-ba26-633c832a77f9-kube-api-access-td2nl\") pod \"cb919ff3-4796-4b2a-ba26-633c832a77f9\" (UID: \"cb919ff3-4796-4b2a-ba26-633c832a77f9\") " Jul 17 22:42:13.667087 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:13.667062 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb919ff3-4796-4b2a-ba26-633c832a77f9-kube-api-access-td2nl" (OuterVolumeSpecName: "kube-api-access-td2nl") pod "cb919ff3-4796-4b2a-ba26-633c832a77f9" (UID: "cb919ff3-4796-4b2a-ba26-633c832a77f9"). InnerVolumeSpecName "kube-api-access-td2nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:42:13.683852 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:13.683821 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7536-139695979215152"] Jul 17 22:42:13.684337 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:13.684318 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb919ff3-4796-4b2a-ba26-633c832a77f9" containerName="test-curl-7536-139695979215152" Jul 17 22:42:13.684429 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:13.684339 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb919ff3-4796-4b2a-ba26-633c832a77f9" containerName="test-curl-7536-139695979215152" Jul 17 22:42:13.684480 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:13.684429 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb919ff3-4796-4b2a-ba26-633c832a77f9" containerName="test-curl-7536-139695979215152" Jul 17 22:42:13.687249 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:13.687230 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979215152" Jul 17 22:42:13.692436 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:13.692406 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7536-139695979215152"] Jul 17 22:42:13.702090 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:13.702057 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb919ff3-4796-4b2a-ba26-633c832a77f9" path="/var/lib/kubelet/pods/cb919ff3-4796-4b2a-ba26-633c832a77f9/volumes" Jul 17 22:42:13.766542 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:13.766464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p844\" (UniqueName: \"kubernetes.io/projected/6eb04b1a-7531-439c-b7fd-8e15471b902f-kube-api-access-4p844\") pod \"test-curl-7536-139695979215152\" (UID: \"6eb04b1a-7531-439c-b7fd-8e15471b902f\") " pod="opendatahub/test-curl-7536-139695979215152" Jul 17 22:42:13.766726 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:13.766669 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td2nl\" (UniqueName: \"kubernetes.io/projected/cb919ff3-4796-4b2a-ba26-633c832a77f9-kube-api-access-td2nl\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:42:13.867881 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:13.867838 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p844\" (UniqueName: \"kubernetes.io/projected/6eb04b1a-7531-439c-b7fd-8e15471b902f-kube-api-access-4p844\") pod \"test-curl-7536-139695979215152\" (UID: \"6eb04b1a-7531-439c-b7fd-8e15471b902f\") " pod="opendatahub/test-curl-7536-139695979215152" Jul 17 22:42:13.876221 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:13.876196 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p844\" (UniqueName: \"kubernetes.io/projected/6eb04b1a-7531-439c-b7fd-8e15471b902f-kube-api-access-4p844\") pod \"test-curl-7536-139695979215152\" (UID: \"6eb04b1a-7531-439c-b7fd-8e15471b902f\") " pod="opendatahub/test-curl-7536-139695979215152" Jul 17 22:42:13.999179 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:13.999139 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979215152" Jul 17 22:42:14.138154 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:14.138130 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7536-139695979215152"] Jul 17 22:42:14.140201 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:42:14.140176 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6eb04b1a_7531_439c_b7fd_8e15471b902f.slice/crio-fd2795913fbc05cf0be8d5daee7311c515d931b2407ab1f8545f87825e75ab99 WatchSource:0}: Error finding container fd2795913fbc05cf0be8d5daee7311c515d931b2407ab1f8545f87825e75ab99: Status 404 returned error can't find the container with id fd2795913fbc05cf0be8d5daee7311c515d931b2407ab1f8545f87825e75ab99 Jul 17 22:42:14.377554 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:14.377456 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7536-139695979215152" event={"ID":"6eb04b1a-7531-439c-b7fd-8e15471b902f","Type":"ContainerStarted","Data":"fd2795913fbc05cf0be8d5daee7311c515d931b2407ab1f8545f87825e75ab99"} Jul 17 22:42:14.378758 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:14.378737 2577 scope.go:117] "RemoveContainer" containerID="551357d0844c3668a6e8a8cb3658b646e465a2ac695bcd01b033fbc0836cd20a" Jul 17 22:42:14.378758 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:14.378755 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979215152" Jul 17 22:42:14.382360 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:14.382330 2577 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7536-139695979215152" oldPodUID="cb919ff3-4796-4b2a-ba26-633c832a77f9" podUID="6eb04b1a-7531-439c-b7fd-8e15471b902f" Jul 17 22:42:15.386955 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:15.386917 2577 generic.go:356] "Generic (PLEG): container finished" podID="6eb04b1a-7531-439c-b7fd-8e15471b902f" containerID="d870ab63c03e4ba9a12a40f47d248c50a07dcb6cd6776ea4127c5facba59fae3" exitCode=0 Jul 17 22:42:15.387411 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:15.387005 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7536-139695979215152" event={"ID":"6eb04b1a-7531-439c-b7fd-8e15471b902f","Type":"ContainerDied","Data":"d870ab63c03e4ba9a12a40f47d248c50a07dcb6cd6776ea4127c5facba59fae3"} Jul 17 22:42:16.532961 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:16.532933 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979215152" Jul 17 22:42:16.559208 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:16.559184 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7536-139695979215152_6eb04b1a-7531-439c-b7fd-8e15471b902f/test-curl-7536-139695979215152/0.log" Jul 17 22:42:16.588761 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:16.588729 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7536-139695979215152"] Jul 17 22:42:16.594506 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:16.594456 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7536-139695979215152"] Jul 17 22:42:16.695798 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:16.695766 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4p844\" (UniqueName: \"kubernetes.io/projected/6eb04b1a-7531-439c-b7fd-8e15471b902f-kube-api-access-4p844\") pod \"6eb04b1a-7531-439c-b7fd-8e15471b902f\" (UID: \"6eb04b1a-7531-439c-b7fd-8e15471b902f\") " Jul 17 22:42:16.698018 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:16.697977 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb04b1a-7531-439c-b7fd-8e15471b902f-kube-api-access-4p844" (OuterVolumeSpecName: "kube-api-access-4p844") pod "6eb04b1a-7531-439c-b7fd-8e15471b902f" (UID: "6eb04b1a-7531-439c-b7fd-8e15471b902f"). InnerVolumeSpecName "kube-api-access-4p844". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 17 22:42:16.797103 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:16.797064 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p844\" (UniqueName: \"kubernetes.io/projected/6eb04b1a-7531-439c-b7fd-8e15471b902f-kube-api-access-4p844\") on node \"ip-10-0-131-14.ec2.internal\" DevicePath \"\"" Jul 17 22:42:17.400515 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:17.400462 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7536-139695979215152" Jul 17 22:42:17.400515 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:17.400482 2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd2795913fbc05cf0be8d5daee7311c515d931b2407ab1f8545f87825e75ab99" Jul 17 22:42:17.411942 ip-10-0-131-14 kubenswrapper[2577]: E0717 22:42:17.411902 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7536-139695979215152\" is forbidden: User \"system:node:ip-10-0-131-14.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-14.ec2.internal' and this object" podUID="6eb04b1a-7531-439c-b7fd-8e15471b902f" pod="opendatahub/test-curl-7536-139695979215152" Jul 17 22:42:17.700516 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:42:17.700464 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eb04b1a-7531-439c-b7fd-8e15471b902f" path="/var/lib/kubelet/pods/6eb04b1a-7531-439c-b7fd-8e15471b902f/volumes" Jul 17 22:43:15.048574 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:15.048478 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6fbf6969b9-dt6sj_1fce1ef7-da7c-4996-a7cd-4780f21286a5/manager/0.log" Jul 17 22:43:15.393466 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:15.393388 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bff9999cf-fr4dc_6cde31f1-f65b-4e27-9d06-8225d0991a44/copy-manifests/0.log" Jul 17 22:43:15.406195 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:15.406171 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bff9999cf-fr4dc_6cde31f1-f65b-4e27-9d06-8225d0991a44/manager/0.log" Jul 17 22:43:16.009748 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:16.009716 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7786b868b8-2zdbb_b41a4c34-1541-4244-a4d7-8f90dae8ed52/maas-api/0.log" Jul 17 22:43:17.602881 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:17.602853 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-qhw5s_c719223e-d7c1-440c-a434-19d91e6df9a6/postgres/0.log" Jul 17 22:43:19.009319 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:19.009287 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-xz9mp_bd9a69c9-2654-44b4-81b5-642ec7af3834/manager/0.log" Jul 17 22:43:19.126161 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:19.126137 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-md9pb_35625c7d-b3f1-44c3-8930-0b2eb0ca5049/manager/0.log" Jul 17 22:43:19.240658 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:19.240631 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-j4lgz_7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c/kuadrant-console-plugin/0.log" Jul 17 22:43:19.359830 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:19.359755 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-4dsh5_eb2ed825-fb18-4db4-8b35-1c3e61fb4d5b/registry-server/0.log" Jul 17 22:43:19.504331 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:19.504298 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8gnqr_d76ef34f-96e7-4272-aff7-71cff43926f5/manager/0.log" Jul 17 22:43:20.673477 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:20.673437 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-nflxn_aadcfd02-e238-4f21-a88e-5481899e72bb/payload-pre-processing/0.log" Jul 17 22:43:20.791169 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:20.791141 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-lr4vw_9157197c-0cda-4d3d-b031-7863f59defe1/payload-processing/0.log" Jul 17 22:43:21.252923 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:21.252890 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-v2cj5_d6bd87f8-9f90-4c08-83a0-a199e45a9114/main/0.log" Jul 17 22:43:21.470209 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:21.470172 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-v49hm_b431df7c-d0d8-48cf-b94e-dbfb026e91af/main/0.log" Jul 17 22:43:27.809574 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:27.809535 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-728z5/must-gather-b56xk"] Jul 17 22:43:27.810013 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:27.809998 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eb04b1a-7531-439c-b7fd-8e15471b902f" containerName="test-curl-7536-139695979215152" Jul 17 22:43:27.810062 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:27.810015 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eb04b1a-7531-439c-b7fd-8e15471b902f" containerName="test-curl-7536-139695979215152" Jul 17 22:43:27.810114 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:27.810105 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eb04b1a-7531-439c-b7fd-8e15471b902f" containerName="test-curl-7536-139695979215152" Jul 17 22:43:27.813473 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:27.813454 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-728z5/must-gather-b56xk" Jul 17 22:43:27.815799 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:27.815775 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-728z5\"/\"kube-root-ca.crt\"" Jul 17 22:43:27.815923 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:27.815804 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-728z5\"/\"default-dockercfg-jpndz\"" Jul 17 22:43:27.815923 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:27.815867 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-728z5\"/\"openshift-service-ca.crt\"" Jul 17 22:43:27.824130 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:27.824111 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-728z5/must-gather-b56xk"] Jul 17 22:43:27.865892 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:27.865862 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km6ns\" (UniqueName: \"kubernetes.io/projected/fe5bb6b1-d838-4483-8c47-99fd7de5a835-kube-api-access-km6ns\") pod \"must-gather-b56xk\" (UID: \"fe5bb6b1-d838-4483-8c47-99fd7de5a835\") " pod="openshift-must-gather-728z5/must-gather-b56xk" Jul 17 22:43:27.866012 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:27.865941 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fe5bb6b1-d838-4483-8c47-99fd7de5a835-must-gather-output\") pod \"must-gather-b56xk\" (UID: \"fe5bb6b1-d838-4483-8c47-99fd7de5a835\") " pod="openshift-must-gather-728z5/must-gather-b56xk" Jul 17 22:43:27.967201 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:27.967169 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fe5bb6b1-d838-4483-8c47-99fd7de5a835-must-gather-output\") pod \"must-gather-b56xk\" (UID: \"fe5bb6b1-d838-4483-8c47-99fd7de5a835\") " pod="openshift-must-gather-728z5/must-gather-b56xk" Jul 17 22:43:27.967353 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:27.967239 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km6ns\" (UniqueName: \"kubernetes.io/projected/fe5bb6b1-d838-4483-8c47-99fd7de5a835-kube-api-access-km6ns\") pod \"must-gather-b56xk\" (UID: \"fe5bb6b1-d838-4483-8c47-99fd7de5a835\") " pod="openshift-must-gather-728z5/must-gather-b56xk" Jul 17 22:43:27.967575 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:27.967555 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/fe5bb6b1-d838-4483-8c47-99fd7de5a835-must-gather-output\") pod \"must-gather-b56xk\" (UID: \"fe5bb6b1-d838-4483-8c47-99fd7de5a835\") " pod="openshift-must-gather-728z5/must-gather-b56xk" Jul 17 22:43:27.976088 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:27.976065 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km6ns\" (UniqueName: \"kubernetes.io/projected/fe5bb6b1-d838-4483-8c47-99fd7de5a835-kube-api-access-km6ns\") pod \"must-gather-b56xk\" (UID: \"fe5bb6b1-d838-4483-8c47-99fd7de5a835\") " pod="openshift-must-gather-728z5/must-gather-b56xk" Jul 17 22:43:28.123744 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:28.123663 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-728z5/must-gather-b56xk" Jul 17 22:43:28.261032 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:28.261011 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-728z5/must-gather-b56xk"] Jul 17 22:43:28.263145 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:43:28.263117 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe5bb6b1_d838_4483_8c47_99fd7de5a835.slice/crio-d6d4aadc68dab461a1aca9ecb6bec9ae29e70b6e3513e29aaccc1400900baef4 WatchSource:0}: Error finding container d6d4aadc68dab461a1aca9ecb6bec9ae29e70b6e3513e29aaccc1400900baef4: Status 404 returned error can't find the container with id d6d4aadc68dab461a1aca9ecb6bec9ae29e70b6e3513e29aaccc1400900baef4 Jul 17 22:43:28.786690 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:28.786656 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-728z5/must-gather-b56xk" event={"ID":"fe5bb6b1-d838-4483-8c47-99fd7de5a835","Type":"ContainerStarted","Data":"d6d4aadc68dab461a1aca9ecb6bec9ae29e70b6e3513e29aaccc1400900baef4"} Jul 17 22:43:29.794851 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:29.794817 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-728z5/must-gather-b56xk" event={"ID":"fe5bb6b1-d838-4483-8c47-99fd7de5a835","Type":"ContainerStarted","Data":"71af65e1b790edf66beca981379397f010835a834eb16c81f234a5cedafe2535"} Jul 17 22:43:29.795233 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:29.794869 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-728z5/must-gather-b56xk" event={"ID":"fe5bb6b1-d838-4483-8c47-99fd7de5a835","Type":"ContainerStarted","Data":"18db6d512cd752c9cc6b4087432071b51dc622b8d4eeca5aa6bbc2566b9bb358"} Jul 17 22:43:29.809327 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:29.809278 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-728z5/must-gather-b56xk" podStartSLOduration=1.746917146 podStartE2EDuration="2.809255731s" totalImagesPullingTime="1.062338585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:43:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:43:29.80847989 +0000 UTC m=+3446.668703387" watchObservedRunningTime="2026-07-17 22:43:29.809255731 +0000 UTC m=+3446.669479239" Jul 17 22:43:30.965382 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:30.965353 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-982gr_e2430562-9210-4afd-9800-9e66f9548cb7/global-pull-secret-syncer/0.log" Jul 17 22:43:31.187701 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:31.187676 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-97mzp_f07efcef-f35f-45b5-873c-28d5acedb9e0/konnectivity-agent/0.log" Jul 17 22:43:31.236568 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:31.236449 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-14.ec2.internal_09f0258f09213a85d398e0f2eb07ae5f/haproxy/0.log" Jul 17 22:43:36.376741 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:36.376706 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-xz9mp_bd9a69c9-2654-44b4-81b5-642ec7af3834/manager/0.log" Jul 17 22:43:36.424881 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:36.424824 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-md9pb_35625c7d-b3f1-44c3-8930-0b2eb0ca5049/manager/0.log" Jul 17 22:43:36.453071 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:36.453040 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-j4lgz_7be84fcb-5e0a-4968-8b8d-1eb9ea1a134c/kuadrant-console-plugin/0.log" Jul 17 22:43:36.511773 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:36.511737 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-4dsh5_eb2ed825-fb18-4db4-8b35-1c3e61fb4d5b/registry-server/0.log" Jul 17 22:43:36.691511 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:36.690964 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8gnqr_d76ef34f-96e7-4272-aff7-71cff43926f5/manager/0.log" Jul 17 22:43:38.285917 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:38.285879 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_284136da-9486-4e1f-a654-ac4459450bbf/alertmanager/0.log" Jul 17 22:43:38.319150 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:38.319121 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_284136da-9486-4e1f-a654-ac4459450bbf/config-reloader/0.log" Jul 17 22:43:38.346249 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:38.346187 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_284136da-9486-4e1f-a654-ac4459450bbf/kube-rbac-proxy-web/0.log" Jul 17 22:43:38.369059 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:38.369032 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_284136da-9486-4e1f-a654-ac4459450bbf/kube-rbac-proxy/0.log" Jul 17 22:43:38.392739 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:38.392698 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_284136da-9486-4e1f-a654-ac4459450bbf/kube-rbac-proxy-metric/0.log" Jul 17 22:43:38.422109 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:38.422086 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_284136da-9486-4e1f-a654-ac4459450bbf/prom-label-proxy/0.log" Jul 17 22:43:38.446297 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:38.446275 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_284136da-9486-4e1f-a654-ac4459450bbf/init-config-reloader/0.log" Jul 17 22:43:38.516759 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:38.516721 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-crzrx_402b046d-053a-4887-a6b9-ec2dbf686caf/kube-state-metrics/0.log" Jul 17 22:43:38.537922 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:38.537848 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-crzrx_402b046d-053a-4887-a6b9-ec2dbf686caf/kube-rbac-proxy-main/0.log" Jul 17 22:43:38.568463 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:38.568425 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-crzrx_402b046d-053a-4887-a6b9-ec2dbf686caf/kube-rbac-proxy-self/0.log" Jul 17 22:43:38.606705 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:38.606681 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-85f79f4d9-w98td_61876aa9-16d9-4016-b24c-f63fa89c32fb/metrics-server/0.log" Jul 17 22:43:38.641475 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:38.641445 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-9ns75_c3f8b396-45d6-403c-a699-9eb5d3d5c627/monitoring-plugin/0.log" Jul 17 22:43:38.690230 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:38.690198 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-24f89_63b9d2a6-f144-44b6-ad72-8f2bdacf512c/node-exporter/0.log" Jul 17 22:43:38.724400 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:38.724376 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-24f89_63b9d2a6-f144-44b6-ad72-8f2bdacf512c/kube-rbac-proxy/0.log" Jul 17 22:43:38.753250 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:38.753225 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-24f89_63b9d2a6-f144-44b6-ad72-8f2bdacf512c/init-textfile/0.log" Jul 17 22:43:38.974507 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:38.974464 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-sw4s8_c1d9d9a0-5aca-4690-9d56-519a5e3791bb/kube-rbac-proxy-main/0.log" Jul 17 22:43:39.011728 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:39.011683 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-sw4s8_c1d9d9a0-5aca-4690-9d56-519a5e3791bb/kube-rbac-proxy-self/0.log" Jul 17 22:43:39.039862 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:39.039829 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-sw4s8_c1d9d9a0-5aca-4690-9d56-519a5e3791bb/openshift-state-metrics/0.log" Jul 17 22:43:39.280966 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:39.280873 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-8wkc7_b2be724c-9805-43fc-a2b4-a131ceefae54/prometheus-operator/0.log" Jul 17 22:43:39.303075 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:39.302995 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-8wkc7_b2be724c-9805-43fc-a2b4-a131ceefae54/kube-rbac-proxy/0.log" Jul 17 22:43:39.331391 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:39.331362 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-r9qbw_783b4ba1-13ca-436f-bf60-81a4f5ff0e24/prometheus-operator-admission-webhook/0.log" Jul 17 22:43:39.368417 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:39.368392 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c54c9c77f-ckk9g_0d19c18f-a526-4fb4-9538-59cd7aa25aa7/telemeter-client/0.log" Jul 17 22:43:39.391731 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:39.391701 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c54c9c77f-ckk9g_0d19c18f-a526-4fb4-9538-59cd7aa25aa7/reload/0.log" Jul 17 22:43:39.413254 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:39.413226 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c54c9c77f-ckk9g_0d19c18f-a526-4fb4-9538-59cd7aa25aa7/kube-rbac-proxy/0.log" Jul 17 22:43:39.453694 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:39.453648 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8666d66855-vl4vv_512670a9-3155-4a10-85c9-d4d4826d32e9/thanos-query/0.log" Jul 17 22:43:39.482037 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:39.482008 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8666d66855-vl4vv_512670a9-3155-4a10-85c9-d4d4826d32e9/kube-rbac-proxy-web/0.log" Jul 17 22:43:39.509995 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:39.509966 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8666d66855-vl4vv_512670a9-3155-4a10-85c9-d4d4826d32e9/kube-rbac-proxy/0.log" Jul 17 22:43:39.533597 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:39.533516 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8666d66855-vl4vv_512670a9-3155-4a10-85c9-d4d4826d32e9/prom-label-proxy/0.log" Jul 17 22:43:39.558280 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:39.558244 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8666d66855-vl4vv_512670a9-3155-4a10-85c9-d4d4826d32e9/kube-rbac-proxy-rules/0.log" Jul 17 22:43:39.581733 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:39.581706 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8666d66855-vl4vv_512670a9-3155-4a10-85c9-d4d4826d32e9/kube-rbac-proxy-metrics/0.log" Jul 17 22:43:41.432637 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.432602 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g"] Jul 17 22:43:41.442611 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.442193 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" Jul 17 22:43:41.445554 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.445526 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g"] Jul 17 22:43:41.517936 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.517898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dd5857f5-b4fe-4de1-9457-c6a4a47e7251-podres\") pod \"perf-node-gather-daemonset-q6h8g\" (UID: \"dd5857f5-b4fe-4de1-9457-c6a4a47e7251\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" Jul 17 22:43:41.518123 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.517952 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd5857f5-b4fe-4de1-9457-c6a4a47e7251-sys\") pod \"perf-node-gather-daemonset-q6h8g\" (UID: \"dd5857f5-b4fe-4de1-9457-c6a4a47e7251\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" Jul 17 22:43:41.518123 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.518066 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dd5857f5-b4fe-4de1-9457-c6a4a47e7251-proc\") pod \"perf-node-gather-daemonset-q6h8g\" (UID: \"dd5857f5-b4fe-4de1-9457-c6a4a47e7251\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" Jul 17 22:43:41.518123 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.518103 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd5857f5-b4fe-4de1-9457-c6a4a47e7251-lib-modules\") pod \"perf-node-gather-daemonset-q6h8g\" (UID: \"dd5857f5-b4fe-4de1-9457-c6a4a47e7251\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" Jul 17 22:43:41.518258 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.518133 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd77h\" (UniqueName: \"kubernetes.io/projected/dd5857f5-b4fe-4de1-9457-c6a4a47e7251-kube-api-access-zd77h\") pod \"perf-node-gather-daemonset-q6h8g\" (UID: \"dd5857f5-b4fe-4de1-9457-c6a4a47e7251\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" Jul 17 22:43:41.619092 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.619057 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd77h\" (UniqueName: \"kubernetes.io/projected/dd5857f5-b4fe-4de1-9457-c6a4a47e7251-kube-api-access-zd77h\") pod \"perf-node-gather-daemonset-q6h8g\" (UID: \"dd5857f5-b4fe-4de1-9457-c6a4a47e7251\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" Jul 17 22:43:41.619286 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.619120 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dd5857f5-b4fe-4de1-9457-c6a4a47e7251-podres\") pod \"perf-node-gather-daemonset-q6h8g\" (UID: \"dd5857f5-b4fe-4de1-9457-c6a4a47e7251\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" Jul 17 22:43:41.619286 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.619156 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd5857f5-b4fe-4de1-9457-c6a4a47e7251-sys\") pod \"perf-node-gather-daemonset-q6h8g\" (UID: \"dd5857f5-b4fe-4de1-9457-c6a4a47e7251\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" Jul 17 22:43:41.619286 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.619225 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dd5857f5-b4fe-4de1-9457-c6a4a47e7251-proc\") pod \"perf-node-gather-daemonset-q6h8g\" (UID: \"dd5857f5-b4fe-4de1-9457-c6a4a47e7251\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" Jul 17 22:43:41.619286 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.619255 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd5857f5-b4fe-4de1-9457-c6a4a47e7251-lib-modules\") pod \"perf-node-gather-daemonset-q6h8g\" (UID: \"dd5857f5-b4fe-4de1-9457-c6a4a47e7251\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" Jul 17 22:43:41.619455 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.619324 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dd5857f5-b4fe-4de1-9457-c6a4a47e7251-podres\") pod \"perf-node-gather-daemonset-q6h8g\" (UID: \"dd5857f5-b4fe-4de1-9457-c6a4a47e7251\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" Jul 17 22:43:41.619455 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.619348 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd5857f5-b4fe-4de1-9457-c6a4a47e7251-sys\") pod \"perf-node-gather-daemonset-q6h8g\" (UID: \"dd5857f5-b4fe-4de1-9457-c6a4a47e7251\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" Jul 17 22:43:41.619455 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.619367 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd5857f5-b4fe-4de1-9457-c6a4a47e7251-lib-modules\") pod \"perf-node-gather-daemonset-q6h8g\" (UID: \"dd5857f5-b4fe-4de1-9457-c6a4a47e7251\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" Jul 17 22:43:41.619455 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.619385 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dd5857f5-b4fe-4de1-9457-c6a4a47e7251-proc\") pod \"perf-node-gather-daemonset-q6h8g\" (UID: \"dd5857f5-b4fe-4de1-9457-c6a4a47e7251\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" Jul 17 22:43:41.628271 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.628239 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd77h\" (UniqueName: \"kubernetes.io/projected/dd5857f5-b4fe-4de1-9457-c6a4a47e7251-kube-api-access-zd77h\") pod \"perf-node-gather-daemonset-q6h8g\" (UID: \"dd5857f5-b4fe-4de1-9457-c6a4a47e7251\") " pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" Jul 17 22:43:41.757954 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.757914 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" Jul 17 22:43:41.949925 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:41.949891 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g"] Jul 17 22:43:41.958698 ip-10-0-131-14 kubenswrapper[2577]: W0717 22:43:41.958667 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddd5857f5_b4fe_4de1_9457_c6a4a47e7251.slice/crio-00ccc1a7bbb09a1b42308cdeaa513c98046d8b60ae47cccfaf630a559be4fdbf WatchSource:0}: Error finding container 00ccc1a7bbb09a1b42308cdeaa513c98046d8b60ae47cccfaf630a559be4fdbf: Status 404 returned error can't find the container with id 00ccc1a7bbb09a1b42308cdeaa513c98046d8b60ae47cccfaf630a559be4fdbf Jul 17 22:43:42.914701 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:42.914661 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" event={"ID":"dd5857f5-b4fe-4de1-9457-c6a4a47e7251","Type":"ContainerStarted","Data":"e311cd11fa50c3eca3d2e13476459e7f61a21f5e1d2dd9094a5e617b57b07337"} Jul 17 22:43:42.915168 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:42.914725 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" event={"ID":"dd5857f5-b4fe-4de1-9457-c6a4a47e7251","Type":"ContainerStarted","Data":"00ccc1a7bbb09a1b42308cdeaa513c98046d8b60ae47cccfaf630a559be4fdbf"} Jul 17 22:43:42.915168 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:42.914882 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" Jul 17 22:43:42.933909 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:42.933853 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" podStartSLOduration=1.933831762 podStartE2EDuration="1.933831762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 22:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:43:42.930527016 +0000 UTC m=+3459.790750539" watchObservedRunningTime="2026-07-17 22:43:42.933831762 +0000 UTC m=+3459.794055264" Jul 17 22:43:43.829466 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:43.829432 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4vnfl_e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed/dns/0.log" Jul 17 22:43:43.851351 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:43.851306 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4vnfl_e1c393ba-2bdd-4e25-9b02-7eb570eaa4ed/kube-rbac-proxy/0.log" Jul 17 22:43:43.995099 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:43.995056 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-j6dqx_55dc68af-baa5-4429-a748-6a5ed468c1b3/dns-node-resolver/0.log" Jul 17 22:43:44.600087 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:44.600061 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-26z9v_2ad8d984-db0a-4d0b-81cf-7c40926cb7b5/node-ca/0.log" Jul 17 22:43:45.955410 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:45.955377 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-nflxn_aadcfd02-e238-4f21-a88e-5481899e72bb/payload-pre-processing/0.log" Jul 17 22:43:45.989279 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:45.989245 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-lr4vw_9157197c-0cda-4d3d-b031-7863f59defe1/payload-processing/0.log" Jul 17 22:43:46.764229 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:46.764188 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4pwvp_67344770-d2f2-44bb-a026-130469fe7114/serve-healthcheck-canary/0.log" Jul 17 22:43:47.528220 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:47.528185 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ntxm7_aac16a59-a485-4da6-8093-78fb71a06bcd/kube-rbac-proxy/0.log" Jul 17 22:43:47.550538 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:47.550511 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ntxm7_aac16a59-a485-4da6-8093-78fb71a06bcd/exporter/0.log" Jul 17 22:43:47.574360 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:47.574299 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ntxm7_aac16a59-a485-4da6-8093-78fb71a06bcd/extractor/0.log" Jul 17 22:43:48.933219 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:48.933186 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-728z5/perf-node-gather-daemonset-q6h8g" Jul 17 22:43:50.267961 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:50.267918 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6fbf6969b9-dt6sj_1fce1ef7-da7c-4996-a7cd-4780f21286a5/manager/0.log" Jul 17 22:43:50.449730 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:50.449697 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bff9999cf-fr4dc_6cde31f1-f65b-4e27-9d06-8225d0991a44/manager/0.log" Jul 17 22:43:50.466587 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:50.466555 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bff9999cf-fr4dc_6cde31f1-f65b-4e27-9d06-8225d0991a44/copy-manifests/0.log" Jul 17 22:43:52.004813 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:52.004787 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-68687b8cf-lcrjt_34b9aba1-2c4a-4440-8a41-c33041695e06/manager/0.log" Jul 17 22:43:52.202614 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:52.202587 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-nflxn_aadcfd02-e238-4f21-a88e-5481899e72bb/payload-pre-processing/0.log" Jul 17 22:43:52.222910 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:52.222884 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-lr4vw_9157197c-0cda-4d3d-b031-7863f59defe1/payload-processing/0.log" Jul 17 22:43:53.734317 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:53.734246 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7677b774bb-nflxn_aadcfd02-e238-4f21-a88e-5481899e72bb/payload-pre-processing/0.log" Jul 17 22:43:53.759342 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:53.759314 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5bf47cd988-lr4vw_9157197c-0cda-4d3d-b031-7863f59defe1/payload-processing/0.log" Jul 17 22:43:58.649204 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:58.649173 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fcfh7_e1ac35ab-4786-4165-8b80-33cd8f78a74b/kube-multus-additional-cni-plugins/0.log" Jul 17 22:43:58.673095 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:58.673066 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fcfh7_e1ac35ab-4786-4165-8b80-33cd8f78a74b/egress-router-binary-copy/0.log" Jul 17 22:43:58.699820 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:58.699789 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fcfh7_e1ac35ab-4786-4165-8b80-33cd8f78a74b/cni-plugins/0.log" Jul 17 22:43:58.728214 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:58.728192 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fcfh7_e1ac35ab-4786-4165-8b80-33cd8f78a74b/bond-cni-plugin/0.log" Jul 17 22:43:58.772186 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:58.772154 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fcfh7_e1ac35ab-4786-4165-8b80-33cd8f78a74b/routeoverride-cni/0.log" Jul 17 22:43:58.818078 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:58.818006 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fcfh7_e1ac35ab-4786-4165-8b80-33cd8f78a74b/whereabouts-cni-bincopy/0.log" Jul 17 22:43:58.865704 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:58.865674 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fcfh7_e1ac35ab-4786-4165-8b80-33cd8f78a74b/whereabouts-cni/0.log" Jul 17 22:43:59.297907 ip-10-0-131-14 kubenswrapper[2577]: I0717 22:43:59.297870 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-kdmxg_cb155f63-69cf-4b8c-ad77-324207fefb52/kube-multus/0.log"