Jul 08 17:38:44.135575 ip-10-0-129-226 systemd[1]: Starting Kubernetes Kubelet... Jul 08 17:38:44.729058 ip-10-0-129-226 kubenswrapper[2584]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 17:38:44.729058 ip-10-0-129-226 kubenswrapper[2584]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 08 17:38:44.729058 ip-10-0-129-226 kubenswrapper[2584]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 17:38:44.729058 ip-10-0-129-226 kubenswrapper[2584]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 08 17:38:44.729058 ip-10-0-129-226 kubenswrapper[2584]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 17:38:44.729058 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.643511 2584 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 08 17:38:44.729058 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651594 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 17:38:44.729058 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651612 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 17:38:44.729058 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651616 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 17:38:44.729058 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651619 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 17:38:44.729058 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651622 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 17:38:44.729058 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651626 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 17:38:44.729058 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651630 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 17:38:44.729058 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651633 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651636 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651639 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651642 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651646 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651649 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651652 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651656 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651659 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651662 2584 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651666 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651670 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651673 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651676 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651679 2584 feature_gate.go:328] unrecognized feature gate: Example Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651682 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651685 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651688 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651691 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651694 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 17:38:44.743494 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651696 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 17:38:44.736379 ip-10-0-129-226 systemd[1]: Started Kubernetes Kubelet. Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651699 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651702 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651704 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651707 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651710 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651713 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651716 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651719 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651721 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651724 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651726 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651729 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651732 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651735 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651738 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651741 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651744 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651747 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651749 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 17:38:44.745803 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651752 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651754 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651757 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651759 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651762 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651764 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651767 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651769 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651772 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651775 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651778 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651780 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651782 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651785 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651787 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651790 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651793 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651795 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651798 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651801 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 17:38:44.747502 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651803 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 17:38:44.759264 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651806 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 17:38:44.759264 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651810 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 17:38:44.759264 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651814 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 17:38:44.759264 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651817 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 17:38:44.759264 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651821 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 17:38:44.759264 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651825 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 17:38:44.759264 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651828 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 17:38:44.759264 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651831 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 17:38:44.759264 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651834 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 17:38:44.759264 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651836 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 17:38:44.759264 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651839 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 17:38:44.759264 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651841 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 17:38:44.759264 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651844 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 17:38:44.759264 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651846 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 17:38:44.759264 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651849 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 17:38:44.759264 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651852 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 17:38:44.759264 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651855 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 17:38:44.759264 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651857 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 17:38:44.759264 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651861 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651864 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651866 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651869 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651871 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651874 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651877 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651879 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651882 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651885 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651888 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651890 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.651893 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652600 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652607 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652610 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652613 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652616 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652618 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652621 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 17:38:44.762370 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652623 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652626 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652629 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652631 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652634 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652637 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652639 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652642 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652645 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652651 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652654 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652656 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652659 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652662 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652664 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652668 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652671 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652674 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652677 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652680 2584 feature_gate.go:328] unrecognized feature gate: Example Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652682 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 17:38:44.775327 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652685 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652687 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652690 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652693 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652696 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652698 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652701 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652703 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652706 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652708 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652710 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652713 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652715 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652718 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652720 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652723 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652725 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652728 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652730 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652733 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 17:38:44.776859 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652737 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 17:38:44.779339 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652740 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 17:38:44.779339 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652747 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 17:38:44.779339 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652750 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 17:38:44.779339 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652755 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 17:38:44.779339 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652759 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 17:38:44.779339 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652763 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 17:38:44.779339 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652766 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 17:38:44.779339 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652769 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 17:38:44.779339 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652772 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 17:38:44.779339 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652775 2584 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 17:38:44.779339 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652777 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 17:38:44.779339 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652780 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 17:38:44.779339 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652783 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 17:38:44.779339 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652786 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 17:38:44.779339 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652788 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 17:38:44.779339 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652791 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 17:38:44.779339 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652794 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 17:38:44.779339 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652796 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 17:38:44.779339 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652799 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 17:38:44.927897 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652802 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 17:38:44.927897 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652804 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 17:38:44.927897 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652807 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 17:38:44.927897 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652810 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 17:38:44.927897 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652814 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 17:38:44.927897 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652817 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 17:38:44.927897 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652819 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 17:38:44.927897 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652822 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 17:38:44.927897 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652825 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 17:38:44.927897 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652827 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 17:38:44.927897 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652830 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 17:38:44.927897 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652834 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 17:38:44.927897 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652839 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 17:38:44.927897 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652842 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 17:38:44.927897 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652845 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 17:38:44.927897 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652849 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 17:38:44.927897 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652853 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 17:38:44.927897 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652856 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 17:38:44.927897 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652860 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 17:38:44.929379 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652862 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 17:38:44.929379 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652865 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 17:38:44.929379 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652868 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 17:38:44.929379 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652871 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 17:38:44.929379 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652874 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 17:38:44.929379 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652876 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 17:38:44.929379 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652879 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 17:38:44.929379 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652882 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 17:38:44.929379 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652884 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 17:38:44.929379 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652887 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 17:38:44.929379 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652889 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 17:38:44.929379 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.652892 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 17:38:44.929379 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653320 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 17:38:44.929379 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653327 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 17:38:44.929379 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653331 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 17:38:44.929379 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653334 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 17:38:44.929379 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653337 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 17:38:44.929379 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653342 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 17:38:44.929379 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653344 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 17:38:44.951893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653347 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 17:38:44.951893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653350 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 17:38:44.951893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653352 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 17:38:44.951893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653355 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 17:38:44.951893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653357 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 17:38:44.951893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653360 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 17:38:44.951893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653364 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 17:38:44.951893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653366 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 17:38:44.951893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653369 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 17:38:44.951893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653372 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 17:38:44.951893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653375 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 17:38:44.951893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653377 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 17:38:44.951893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653380 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 17:38:44.951893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653382 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 17:38:44.951893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653385 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 17:38:44.951893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653388 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 17:38:44.951893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653390 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 17:38:44.951893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653393 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 17:38:44.951893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653396 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653398 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653400 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653403 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653405 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653408 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653411 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653413 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653416 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653419 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653421 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653424 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653428 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653431 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653434 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653436 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653439 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653441 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653443 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653446 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653450 2584 feature_gate.go:328] unrecognized feature gate: Example Jul 08 17:38:44.954025 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653452 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653455 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653457 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653460 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653462 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653465 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653467 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653470 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653472 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653475 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653477 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653481 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653485 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653487 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653491 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653494 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653496 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653498 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653501 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653503 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 17:38:44.955114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653506 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 17:38:44.956163 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653508 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 17:38:44.956163 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653511 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 17:38:44.956163 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653515 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 17:38:44.956163 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653519 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 17:38:44.956163 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653523 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 17:38:44.956163 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653525 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 17:38:44.956163 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653528 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 17:38:44.956163 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653531 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 17:38:44.956163 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653534 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 17:38:44.956163 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653536 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 17:38:44.956163 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653540 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 17:38:44.956163 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653542 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 17:38:44.956163 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653545 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 17:38:44.956163 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653548 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 17:38:44.956163 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653550 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 17:38:44.956163 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653553 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 17:38:44.956163 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653555 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 17:38:44.956163 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653558 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 17:38:44.956163 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653561 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653565 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653568 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653570 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653573 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653575 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653578 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653580 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653582 2584 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653585 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653587 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653590 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653592 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.653594 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653678 2584 flags.go:64] FLAG: --address="0.0.0.0" Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653694 2584 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653703 2584 flags.go:64] FLAG: --anonymous-auth="true" Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653709 2584 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653715 2584 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653718 2584 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653723 2584 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 08 17:38:44.957196 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653728 2584 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653731 2584 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653734 2584 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653739 2584 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653743 2584 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653746 2584 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653749 2584 flags.go:64] FLAG: --cgroup-root="" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653752 2584 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653755 2584 flags.go:64] FLAG: --client-ca-file="" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653758 2584 flags.go:64] FLAG: --cloud-provider="external" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653761 2584 flags.go:64] FLAG: --cluster-dns="[]" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653765 2584 flags.go:64] FLAG: --cluster-domain="" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653768 2584 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653771 2584 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653774 2584 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653777 2584 flags.go:64] FLAG: --container-log-max-files="5" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653781 2584 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653784 2584 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653787 2584 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653790 2584 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653793 2584 flags.go:64] FLAG: --contention-profiling="false" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653795 2584 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653799 2584 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653802 2584 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 08 17:38:44.957796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653804 2584 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653808 2584 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653811 2584 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653816 2584 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653818 2584 flags.go:64] FLAG: --enable-load-reader="false" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653821 2584 flags.go:64] FLAG: --enable-server="true" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653824 2584 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653829 2584 flags.go:64] FLAG: --event-burst="100" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653832 2584 flags.go:64] FLAG: --event-qps="50" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653835 2584 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653838 2584 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653842 2584 flags.go:64] FLAG: --eviction-hard="" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653846 2584 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653849 2584 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653852 2584 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653855 2584 flags.go:64] FLAG: --eviction-soft="" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653858 2584 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653861 2584 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653864 2584 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653867 2584 flags.go:64] FLAG: --experimental-mounter-path="" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653870 2584 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653873 2584 flags.go:64] FLAG: --fail-swap-on="true" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653876 2584 flags.go:64] FLAG: --feature-gates="" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653882 2584 flags.go:64] FLAG: --file-check-frequency="20s" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653885 2584 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 08 17:38:44.958618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653888 2584 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653891 2584 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653894 2584 flags.go:64] FLAG: --healthz-port="10248" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653897 2584 flags.go:64] FLAG: --help="false" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653901 2584 flags.go:64] FLAG: --hostname-override="ip-10-0-129-226.ec2.internal" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653904 2584 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653907 2584 flags.go:64] FLAG: --http-check-frequency="20s" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653910 2584 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653913 2584 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653916 2584 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653921 2584 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653924 2584 flags.go:64] FLAG: --image-service-endpoint="" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653927 2584 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653929 2584 flags.go:64] FLAG: --kube-api-burst="100" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653932 2584 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653935 2584 flags.go:64] FLAG: --kube-api-qps="50" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653938 2584 flags.go:64] FLAG: --kube-reserved="" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653941 2584 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653945 2584 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653948 2584 flags.go:64] FLAG: --kubelet-cgroups="" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653951 2584 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653954 2584 flags.go:64] FLAG: --lock-file="" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653956 2584 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653959 2584 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 08 17:38:44.959442 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653962 2584 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653967 2584 flags.go:64] FLAG: --log-json-split-stream="false" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653974 2584 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653977 2584 flags.go:64] FLAG: --log-text-split-stream="false" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653979 2584 flags.go:64] FLAG: --logging-format="text" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653982 2584 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653985 2584 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653988 2584 flags.go:64] FLAG: --manifest-url="" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653991 2584 flags.go:64] FLAG: --manifest-url-header="" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653995 2584 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.653998 2584 flags.go:64] FLAG: --max-open-files="1000000" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654003 2584 flags.go:64] FLAG: --max-pods="110" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654006 2584 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654009 2584 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654011 2584 flags.go:64] FLAG: --memory-manager-policy="None" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654014 2584 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654017 2584 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654020 2584 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654025 2584 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654032 2584 flags.go:64] FLAG: --node-status-max-images="50" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654035 2584 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654038 2584 flags.go:64] FLAG: --oom-score-adj="-999" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654041 2584 flags.go:64] FLAG: --pod-cidr="" Jul 08 17:38:44.960287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654044 2584 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654049 2584 flags.go:64] FLAG: --pod-manifest-path="" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654051 2584 flags.go:64] FLAG: --pod-max-pids="-1" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654056 2584 flags.go:64] FLAG: --pods-per-core="0" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654059 2584 flags.go:64] FLAG: --port="10250" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654062 2584 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654064 2584 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0db12d02f9252a9c7" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654068 2584 flags.go:64] FLAG: --qos-reserved="" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654071 2584 flags.go:64] FLAG: --read-only-port="10255" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654074 2584 flags.go:64] FLAG: --register-node="true" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654077 2584 flags.go:64] FLAG: --register-with-taints="" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654080 2584 flags.go:64] FLAG: --registry-burst="10" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654096 2584 flags.go:64] FLAG: --registry-qps="5" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654100 2584 flags.go:64] FLAG: --reserved-cpus="" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654103 2584 flags.go:64] FLAG: --reserved-memory="" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654107 2584 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654110 2584 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654113 2584 flags.go:64] FLAG: --rotate-certificates="false" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654116 2584 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654119 2584 flags.go:64] FLAG: --runonce="false" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654122 2584 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654125 2584 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654128 2584 flags.go:64] FLAG: --seccomp-default="false" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654131 2584 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654133 2584 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654137 2584 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 08 17:38:44.962364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654139 2584 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654142 2584 flags.go:64] FLAG: --storage-driver-password="root" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654147 2584 flags.go:64] FLAG: --storage-driver-secure="false" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654149 2584 flags.go:64] FLAG: --storage-driver-table="stats" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654152 2584 flags.go:64] FLAG: --storage-driver-user="root" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654155 2584 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654158 2584 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654160 2584 flags.go:64] FLAG: --system-cgroups="" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654163 2584 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654170 2584 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654172 2584 flags.go:64] FLAG: --tls-cert-file="" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654175 2584 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654179 2584 flags.go:64] FLAG: --tls-min-version="" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654182 2584 flags.go:64] FLAG: --tls-private-key-file="" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654185 2584 flags.go:64] FLAG: --topology-manager-policy="none" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654188 2584 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654190 2584 flags.go:64] FLAG: --topology-manager-scope="container" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654193 2584 flags.go:64] FLAG: --v="2" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654197 2584 flags.go:64] FLAG: --version="false" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654202 2584 flags.go:64] FLAG: --vmodule="" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654206 2584 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.654209 2584 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654392 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654397 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 17:38:44.963124 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654400 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 17:38:45.108587 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654405 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 17:38:45.108587 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654409 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 17:38:45.108587 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654412 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 17:38:45.108587 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654415 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 17:38:45.108587 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654418 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 17:38:45.108587 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654420 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 17:38:45.108587 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654423 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 17:38:45.108587 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654426 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 17:38:45.108587 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654428 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 17:38:45.108587 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654433 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 17:38:45.108587 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654435 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 17:38:45.108587 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654438 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 17:38:45.108587 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654441 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 17:38:45.108587 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654444 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 17:38:45.108587 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654446 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 17:38:45.108587 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654449 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 17:38:45.108587 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654453 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 17:38:45.108587 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654456 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 17:38:45.108587 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654458 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 17:38:45.109303 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654461 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 17:38:45.109303 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654463 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 17:38:45.109303 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654466 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 17:38:45.109303 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654468 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 17:38:45.109303 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654472 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 17:38:45.109303 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654475 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 17:38:45.109303 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654478 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 17:38:45.109303 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654481 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 17:38:45.109303 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654490 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 17:38:45.109303 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654493 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 17:38:45.109303 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654496 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 17:38:45.109303 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654498 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 17:38:45.109303 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654501 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 17:38:45.109303 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654504 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 17:38:45.109303 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654506 2584 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 17:38:45.109303 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654509 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 17:38:45.109303 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654511 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 17:38:45.109303 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654514 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 17:38:45.109303 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654516 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654519 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654521 2584 feature_gate.go:328] unrecognized feature gate: Example Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654525 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654530 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654532 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654535 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654538 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654540 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654543 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654546 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654549 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654552 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654554 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654557 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654560 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654562 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654565 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654567 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654569 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 17:38:45.109887 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654572 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654574 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654577 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654585 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654587 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654590 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654592 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654595 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654598 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654600 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654602 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654605 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654608 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654610 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654612 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654615 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654619 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654621 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654624 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654627 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 17:38:45.110530 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654629 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 17:38:45.111078 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654632 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 17:38:45.111078 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654635 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 17:38:45.111078 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654638 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 17:38:45.111078 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654641 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 17:38:45.111078 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654644 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 17:38:45.111078 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654646 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 17:38:45.111078 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654649 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 17:38:45.111078 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654651 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 17:38:45.111078 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654654 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 17:38:45.111078 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654656 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 17:38:45.111078 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654659 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 17:38:45.111078 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654662 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 17:38:45.111078 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654665 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 17:38:45.111078 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654668 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 17:38:45.111078 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654670 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 17:38:45.111078 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654673 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 17:38:45.111078 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654676 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 17:38:45.111078 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.654678 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 17:38:45.111677 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.655656 2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 17:38:45.111677 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.663385 2584 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 08 17:38:45.111677 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.663407 2584 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 08 17:38:45.111677 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.664345 2584 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 08 17:38:45.111677 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.664356 2584 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 08 17:38:45.111677 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664432 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 17:38:45.111677 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664437 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 17:38:45.111677 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664440 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 17:38:45.111677 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664444 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 17:38:45.111677 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664447 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 17:38:45.111677 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664450 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 17:38:45.111677 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664452 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 17:38:45.111677 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664455 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 17:38:45.111677 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664457 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 17:38:45.111677 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664460 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 17:38:45.112102 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664463 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 17:38:45.112102 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664466 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 17:38:45.112102 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664468 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 17:38:45.112102 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664471 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 17:38:45.112102 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664474 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 17:38:45.112102 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664476 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 17:38:45.112102 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664480 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 17:38:45.112102 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664483 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 17:38:45.112102 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664486 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 17:38:45.112102 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664488 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 17:38:45.112102 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664491 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 17:38:45.112102 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664494 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 17:38:45.112102 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664496 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 17:38:45.112102 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664499 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 17:38:45.112102 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664502 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 17:38:45.112102 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664504 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 17:38:45.112102 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664507 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 17:38:45.112102 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664509 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 17:38:45.112102 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664512 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664516 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664519 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664522 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664524 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664527 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664529 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664532 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664538 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664542 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664545 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664547 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664550 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664552 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664555 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664557 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664560 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664563 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664565 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664568 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 17:38:45.112701 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664570 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 17:38:45.113257 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664573 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 17:38:45.113257 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664576 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 17:38:45.113257 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664578 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 17:38:45.113257 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664581 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 17:38:45.113257 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664583 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 17:38:45.113257 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664587 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 17:38:45.113257 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664591 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 17:38:45.113257 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664594 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 17:38:45.113257 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664597 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 17:38:45.113257 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664600 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 17:38:45.113257 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664602 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 17:38:45.113257 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664605 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 17:38:45.113257 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664607 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 17:38:45.113257 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664611 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 17:38:45.113257 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664613 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 17:38:45.113257 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664616 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 17:38:45.113257 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664618 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 17:38:45.113257 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664621 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 17:38:45.113257 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664624 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664628 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664631 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664634 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664636 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664639 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664642 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664644 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664646 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664649 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664651 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664654 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664657 2584 feature_gate.go:328] unrecognized feature gate: Example Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664659 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664662 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664664 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664666 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664669 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664671 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664675 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 17:38:45.113842 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664679 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 17:38:45.114644 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664682 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 17:38:45.114644 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664686 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 17:38:45.114644 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664688 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 17:38:45.114644 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664691 2584 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 17:38:45.114644 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664695 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 17:38:45.114644 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664698 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 17:38:45.114644 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664701 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 17:38:45.114644 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664705 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 17:38:45.114644 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664708 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 17:38:45.114644 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664710 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 17:38:45.114644 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664713 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 17:38:45.114644 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.664718 2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 17:38:45.114644 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664848 2584 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 17:38:45.114644 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664855 2584 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 17:38:45.114644 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664857 2584 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 17:38:45.115513 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664860 2584 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 17:38:45.115513 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664863 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 17:38:45.115513 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664866 2584 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 17:38:45.115513 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664868 2584 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 17:38:45.115513 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664872 2584 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 17:38:45.115513 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664876 2584 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 17:38:45.115513 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664879 2584 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 17:38:45.115513 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664882 2584 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 17:38:45.115513 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664884 2584 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 17:38:45.115513 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664887 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 17:38:45.115513 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664890 2584 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 17:38:45.115513 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664892 2584 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 17:38:45.115513 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664895 2584 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 17:38:45.115513 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664898 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 17:38:45.115513 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664900 2584 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 17:38:45.115513 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664903 2584 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 17:38:45.115513 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664905 2584 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 17:38:45.115513 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664908 2584 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 17:38:45.115513 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664910 2584 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 17:38:45.116551 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664914 2584 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 17:38:45.116551 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664917 2584 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 17:38:45.116551 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664920 2584 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 17:38:45.116551 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664923 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 17:38:45.116551 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664926 2584 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 17:38:45.116551 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664929 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 17:38:45.116551 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664932 2584 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 17:38:45.116551 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664935 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 17:38:45.116551 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664937 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 17:38:45.116551 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664940 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 17:38:45.116551 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664942 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 17:38:45.116551 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664948 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 17:38:45.116551 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664950 2584 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 17:38:45.116551 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664953 2584 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 17:38:45.116551 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664956 2584 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 17:38:45.116551 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664958 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 17:38:45.116551 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664961 2584 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 17:38:45.116551 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664964 2584 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 17:38:45.116551 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664966 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 17:38:45.117171 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664969 2584 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 17:38:45.117171 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664971 2584 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 17:38:45.117171 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664974 2584 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 17:38:45.117171 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664976 2584 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 17:38:45.117171 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664979 2584 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 17:38:45.117171 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664982 2584 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 17:38:45.117171 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664984 2584 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 17:38:45.117171 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664987 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 17:38:45.117171 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664989 2584 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 17:38:45.117171 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664991 2584 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 17:38:45.117171 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664994 2584 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 17:38:45.117171 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.664996 2584 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 17:38:45.117171 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665000 2584 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 17:38:45.117171 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665003 2584 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 17:38:45.117171 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665005 2584 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 17:38:45.117171 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665008 2584 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 17:38:45.117171 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665011 2584 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 17:38:45.117171 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665014 2584 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 17:38:45.117171 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665016 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665019 2584 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665022 2584 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665024 2584 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665026 2584 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665029 2584 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665034 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665037 2584 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665039 2584 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665042 2584 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665044 2584 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665047 2584 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665050 2584 feature_gate.go:328] unrecognized feature gate: Example Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665052 2584 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665055 2584 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665057 2584 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665060 2584 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665062 2584 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665064 2584 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665067 2584 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 17:38:45.117798 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665070 2584 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 17:38:45.118444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665072 2584 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 17:38:45.118444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665075 2584 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 17:38:45.118444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665077 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 17:38:45.118444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665080 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 17:38:45.118444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665082 2584 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 17:38:45.118444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665085 2584 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 17:38:45.118444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665087 2584 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 17:38:45.118444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665089 2584 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 17:38:45.118444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665092 2584 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 17:38:45.118444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665094 2584 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 17:38:45.118444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665097 2584 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 17:38:45.118444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665099 2584 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 17:38:45.118444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665101 2584 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 17:38:45.118444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665104 2584 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 17:38:45.118444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665107 2584 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 17:38:45.118444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665109 2584 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 17:38:45.118444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665112 2584 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 17:38:45.118444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665117 2584 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 17:38:45.118444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:44.665119 2584 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 17:38:45.119063 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.665124 2584 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 17:38:45.119063 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.665281 2584 server.go:962] "Client rotation is on, will bootstrap in background" Jul 08 17:38:45.119063 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.671839 2584 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 08 17:38:45.119063 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.673561 2584 server.go:1019] "Starting client certificate rotation" Jul 08 17:38:45.119063 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.673673 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 17:38:45.119063 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.673711 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 17:38:45.119063 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.678361 2584 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 17:38:45.119063 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.684270 2584 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 17:38:45.119063 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.699555 2584 log.go:25] "Validated CRI v1 runtime API" Jul 08 17:38:45.119063 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.702125 2584 log.go:25] "Validated CRI v1 image API" Jul 08 17:38:45.119063 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.702955 2584 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 08 17:38:45.119063 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.704636 2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 17:38:45.119482 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.708010 2584 fs.go:135] Filesystem UUIDs: map[1680477e-e921-4f82-9921-ef4d6dff939a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9a9aa119-820a-4e24-9180-2dbb199ae3ef:/dev/nvme0n1p4] Jul 08 17:38:45.119482 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.708026 2584 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 08 17:38:45.119898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.712345 2584 manager.go:216] Machine: {Timestamp:2026-07-08 17:38:44.71182721 +0000 UTC m=+0.395924478 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098704 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec249547fd9736cff9000323d3a69836 SystemUUID:ec249547-fd97-36cf-f900-0323d3a69836 BootID:b0a34606-031b-45c7-bad2-ac61810c8a99 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f3:9a:54:4d:b7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f3:9a:54:4d:b7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:67:ba:cb:15:c7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 08 17:38:45.119898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.712449 2584 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 08 17:38:45.119898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.712537 2584 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 08 17:38:45.119898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.716047 2584 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 08 17:38:45.119898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.716076 2584 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-226.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 08 17:38:45.119898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.716886 2584 topology_manager.go:138] "Creating topology manager with none policy" Jul 08 17:38:45.119898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.716896 2584 container_manager_linux.go:309] "Creating device plugin manager" Jul 08 17:38:45.119898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.716906 2584 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 17:38:45.119898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.716928 2584 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 17:38:45.119898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.718863 2584 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 08 17:38:45.119898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.723608 2584 state_mem.go:36] "Initialized new in-memory state store" Jul 08 17:38:45.119898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.723751 2584 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 08 17:38:45.119898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.723782 2584 kubelet.go:486] "Attempting to sync node with API server" Jul 08 17:38:45.119898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.723792 2584 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 08 17:38:45.119898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.723806 2584 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 08 17:38:45.119898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.723817 2584 kubelet.go:398] "Adding apiserver pod source" Jul 08 17:38:45.119898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.723826 2584 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 08 17:38:45.119898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.724292 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.724309 2584 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.726846 2584 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.729316 2584 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.729344 2584 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.730842 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.730855 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.730861 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.730866 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.730875 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.730880 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.730888 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.730893 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.730900 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.730905 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.730914 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.730920 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.731826 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.731835 2584 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.735498 2584 server.go:1268] "Started kubelet" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.735541 2584 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.735620 2584 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.735692 2584 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:44.736061 2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.736114 2584 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-226.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:44.736502 2584 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.738886 2584 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.739771 2584 server.go:310] "Adding debug handlers to kubelet server" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:44.741845 2584 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:44.742381 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-226.ec2.internal.18c061332544e1c3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-226.ec2.internal,UID:ip-10-0-129-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-226.ec2.internal,},FirstTimestamp:2026-07-08 17:38:44.735214019 +0000 UTC m=+0.419311294,LastTimestamp:2026-07-08 17:38:44.735214019 +0000 UTC m=+0.419311294,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-226.ec2.internal,}" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.747806 2584 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.747821 2584 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:44.747916 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.748658 2584 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.748765 2584 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.748779 2584 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.748897 2584 reconstruct.go:97] "Volume reconstruction finished" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.748903 2584 reconciler.go:29] "Reconciler: start to sync state" Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.749221 2584 factory.go:153] Registering CRI-O factory Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.749234 2584 factory.go:223] Registration of the crio container factory successfully Jul 08 17:38:45.121491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.749287 2584 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.749295 2584 factory.go:55] Registering systemd factory Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.749301 2584 factory.go:223] Registration of the systemd container factory successfully Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.749320 2584 factory.go:103] Registering Raw factory Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.749329 2584 manager.go:1196] Started watching for new ooms in manager Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.749617 2584 manager.go:349] Starting recovery of all containers Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:44.757890 2584 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-226.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:44.757950 2584 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.760539 2584 manager.go:354] Recovery completed Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.765625 2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mnsdd" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.767414 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.769589 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.769613 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.769623 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.770122 2584 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.770131 2584 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.770133 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.770146 2584 state_mem.go:36] "Initialized new in-memory state store" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.770829 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.770851 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.770861 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:44.771585 2584 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-226.ec2.internal.18c0613327519355 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-226.ec2.internal,UID:ip-10-0-129-226.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-226.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-226.ec2.internal,},FirstTimestamp:2026-07-08 17:38:44.769600341 +0000 UTC m=+0.453697609,LastTimestamp:2026-07-08 17:38:44.769600341 +0000 UTC m=+0.453697609,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-226.ec2.internal,}" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.772358 2584 policy_none.go:49] "None policy: Start" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.772372 2584 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.772382 2584 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.774342 2584 policy_none.go:47] "Start" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.774502 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mnsdd" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.810009 2584 manager.go:337] "Starting Device Plugin manager" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:44.810049 2584 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.810061 2584 server.go:87] "Starting device plugin registration server" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.810379 2584 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.810393 2584 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.810488 2584 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.810577 2584 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.810589 2584 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:44.811137 2584 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:44.811194 2584 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-226.ec2.internal\" not found" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.824690 2584 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2616 into the system container \"/system.slice\": " Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.911067 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:38:45.122614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.912359 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.912385 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.912396 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.912424 2584 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-226.ec2.internal" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.920074 2584 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-226.ec2.internal" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:44.920096 2584 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-226.ec2.internal\": node \"ip-10-0-129-226.ec2.internal\" not found" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:44.945645 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.948262 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.949551 2584 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.949596 2584 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.949626 2584 kubelet.go:2448] "Starting kubelet main sync loop" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:44.949664 2584 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:44.951683 2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:45.046370 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.050672 2584 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal"] Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.050739 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.055496 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.055524 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.055536 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.057900 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.058020 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.058054 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.058642 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.058648 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.058666 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.058672 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.058677 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.058684 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.060800 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.060820 2584 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.061618 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientMemory" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.061645 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.061654 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeHasSufficientPID" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:45.101367 2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-226.ec2.internal\" not found" node="ip-10-0-129-226.ec2.internal" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:45.106213 2584 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-226.ec2.internal\" not found" node="ip-10-0-129-226.ec2.internal" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:45.147886 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found" Jul 08 17:38:45.256338 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.151020 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8119fcfc2e17d9ba7073558077e669dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal\" (UID: \"8119fcfc2e17d9ba7073558077e669dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal" Jul 08 17:38:45.257417 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.151042 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/66beb382798cae7e66e45d75b5f0b9d7-config\") pod \"kube-apiserver-proxy-ip-10-0-129-226.ec2.internal\" (UID: \"66beb382798cae7e66e45d75b5f0b9d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal" Jul 08 17:38:45.257417 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.151059 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8119fcfc2e17d9ba7073558077e669dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal\" (UID: \"8119fcfc2e17d9ba7073558077e669dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal" Jul 08 17:38:45.257417 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:45.248626 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found" Jul 08 17:38:45.257417 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.251892 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8119fcfc2e17d9ba7073558077e669dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal\" (UID: \"8119fcfc2e17d9ba7073558077e669dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal" Jul 08 17:38:45.257417 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.251922 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8119fcfc2e17d9ba7073558077e669dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal\" (UID: \"8119fcfc2e17d9ba7073558077e669dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal" Jul 08 17:38:45.257417 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.251942 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/66beb382798cae7e66e45d75b5f0b9d7-config\") pod \"kube-apiserver-proxy-ip-10-0-129-226.ec2.internal\" (UID: \"66beb382798cae7e66e45d75b5f0b9d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal" Jul 08 17:38:45.257417 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.251999 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/66beb382798cae7e66e45d75b5f0b9d7-config\") pod \"kube-apiserver-proxy-ip-10-0-129-226.ec2.internal\" (UID: \"66beb382798cae7e66e45d75b5f0b9d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal" Jul 08 17:38:45.257417 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.252004 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8119fcfc2e17d9ba7073558077e669dc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal\" (UID: \"8119fcfc2e17d9ba7073558077e669dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal" Jul 08 17:38:45.257417 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.252014 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8119fcfc2e17d9ba7073558077e669dc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal\" (UID: \"8119fcfc2e17d9ba7073558077e669dc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal" Jul 08 17:38:45.389948 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:45.349134 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found" Jul 08 17:38:45.507981 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.405439 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal" Jul 08 17:38:45.507981 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.410738 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal" Jul 08 17:38:45.507981 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:45.449353 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found" Jul 08 17:38:45.687014 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:45.551017 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found" Jul 08 17:38:45.687014 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:45.652766 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found" Jul 08 17:38:45.687014 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.673530 2584 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 08 17:38:45.687014 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.673741 2584 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 17:38:45.846850 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:45.753057 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found" Jul 08 17:38:45.846850 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.777786 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 17:33:44 +0000 UTC" deadline="2028-03-08 23:37:43.222134298 +0000 UTC" Jul 08 17:38:45.846850 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.777837 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14621h58m57.444302893s" Jul 08 17:38:45.846850 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.817526 2584 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 17:38:45.978247 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:45.853222 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found" Jul 08 17:38:45.978247 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:45.953431 2584 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-226.ec2.internal\" not found" Jul 08 17:38:45.978247 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:45.975664 2584 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 17:38:46.155330 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.048307 2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal" Jul 08 17:38:46.155330 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.064081 2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 17:38:46.155330 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.066351 2584 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal" Jul 08 17:38:46.155330 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.080715 2584 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 17:38:46.283694 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.210184 2584 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 17:38:46.592198 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:46.468076 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8119fcfc2e17d9ba7073558077e669dc.slice/crio-2ddc5f50edd2386a670db1996d9a0aa446e292f8643a9b456ad8533651358184 WatchSource:0}: Error finding container 2ddc5f50edd2386a670db1996d9a0aa446e292f8643a9b456ad8533651358184: Status 404 returned error can't find the container with id 2ddc5f50edd2386a670db1996d9a0aa446e292f8643a9b456ad8533651358184 Jul 08 17:38:46.592198 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.477528 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 17:38:46.592198 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:46.479064 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66beb382798cae7e66e45d75b5f0b9d7.slice/crio-1c935fe7eea87113a6c5e94961ce9ee9f02d6d68db6efdda6482fbbb6a6d36c3 WatchSource:0}: Error finding container 1c935fe7eea87113a6c5e94961ce9ee9f02d6d68db6efdda6482fbbb6a6d36c3: Status 404 returned error can't find the container with id 1c935fe7eea87113a6c5e94961ce9ee9f02d6d68db6efdda6482fbbb6a6d36c3 Jul 08 17:38:46.742933 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.727670 2584 apiserver.go:52] "Watching apiserver" Jul 08 17:38:46.742933 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.737422 2584 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 17:38:46.742933 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.738790 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp","openshift-cluster-node-tuning-operator/tuned-4zlvf","openshift-image-registry/node-ca-rhlxp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal","openshift-multus/multus-8zt5s","openshift-multus/multus-additional-cni-plugins-blvkt","kube-system/global-pull-secret-syncer-gbq5c","openshift-dns/node-resolver-8h4h6","openshift-multus/network-metrics-daemon-h6mc5","openshift-network-diagnostics/network-check-target-d6xks","openshift-network-operator/iptables-alerter-95k7k","openshift-ovn-kubernetes/ovnkube-node-krx82","kube-system/konnectivity-agent-bl792","kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal"] Jul 08 17:38:46.742933 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.742226 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:38:46.742933 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:46.742315 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:38:46.745033 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.744397 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bl792" Jul 08 17:38:46.747598 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.746706 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rhlxp" Jul 08 17:38:46.747598 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.746765 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 17:38:46.747598 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.746979 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 17:38:46.747598 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.746995 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7kr5g\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.748724 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.748795 2584 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.749032 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.749096 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6tp2n\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.749377 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8h4h6" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.749393 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.751466 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.751831 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8zt5s" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.752365 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.752395 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-c2ft6\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.754250 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.754269 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ncdjm\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.754501 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.754501 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.754574 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.754637 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.757179 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.757222 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.757239 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-25c4c\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:46.757273 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.757294 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.759919 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:46.759978 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.761962 2584 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.762589 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.765035 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.765223 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.765304 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gcscn\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.765440 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.767332 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6fk94\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.767437 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.767617 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-95k7k" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.767701 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.767870 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 17:38:46.908376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.770306 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rj2dn\"" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.770363 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.771031 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.771064 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.777916 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.780317 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4zfs8\"" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.780417 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.780462 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.780658 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.781866 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/970dfcb3-0c92-4a96-95c9-fe1db4c711d6-agent-certs\") pod \"konnectivity-agent-bl792\" (UID: \"970dfcb3-0c92-4a96-95c9-fe1db4c711d6\") " pod="kube-system/konnectivity-agent-bl792" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.781907 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/970dfcb3-0c92-4a96-95c9-fe1db4c711d6-konnectivity-ca\") pod \"konnectivity-agent-bl792\" (UID: \"970dfcb3-0c92-4a96-95c9-fe1db4c711d6\") " pod="kube-system/konnectivity-agent-bl792" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.781936 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/747083d6-492d-413c-99e7-364aca1c1cee-host\") pod \"node-ca-rhlxp\" (UID: \"747083d6-492d-413c-99e7-364aca1c1cee\") " pod="openshift-image-registry/node-ca-rhlxp" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.781965 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4ksh\" (UniqueName: \"kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh\") pod \"network-check-target-d6xks\" (UID: \"135de3f6-fc59-4070-9899-88ac1d14ffde\") " pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.781993 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/747083d6-492d-413c-99e7-364aca1c1cee-serviceca\") pod \"node-ca-rhlxp\" (UID: \"747083d6-492d-413c-99e7-364aca1c1cee\") " pod="openshift-image-registry/node-ca-rhlxp" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.782024 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q76x9\" (UniqueName: \"kubernetes.io/projected/747083d6-492d-413c-99e7-364aca1c1cee-kube-api-access-q76x9\") pod \"node-ca-rhlxp\" (UID: \"747083d6-492d-413c-99e7-364aca1c1cee\") " pod="openshift-image-registry/node-ca-rhlxp" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.782054 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e8468fb3-5642-43d6-95e5-c524fb532087-hosts-file\") pod \"node-resolver-8h4h6\" (UID: \"e8468fb3-5642-43d6-95e5-c524fb532087\") " pod="openshift-dns/node-resolver-8h4h6" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.782078 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8468fb3-5642-43d6-95e5-c524fb532087-tmp-dir\") pod \"node-resolver-8h4h6\" (UID: \"e8468fb3-5642-43d6-95e5-c524fb532087\") " pod="openshift-dns/node-resolver-8h4h6" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.782103 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jqcn\" (UniqueName: \"kubernetes.io/projected/e8468fb3-5642-43d6-95e5-c524fb532087-kube-api-access-7jqcn\") pod \"node-resolver-8h4h6\" (UID: \"e8468fb3-5642-43d6-95e5-c524fb532087\") " pod="openshift-dns/node-resolver-8h4h6" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.782163 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.782656 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.782827 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.791990 2584 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4n2nw" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.798380 2584 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4n2nw" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.849312 2584 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882276 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b773719-d2d4-4be0-a119-71ffc944336c-ovnkube-script-lib\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.910041 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882310 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-sysconfig\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:46.911748 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882329 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a8c9e0e-553f-4e86-af3a-0277d352166c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:46.911748 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882348 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b773719-d2d4-4be0-a119-71ffc944336c-ovn-node-metrics-cert\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.911748 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882387 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-sysctl-conf\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:46.911748 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882411 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-multus-conf-dir\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:46.911748 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882428 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-multus-cni-dir\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:46.911748 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882469 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4e9d179-2384-44e7-afd8-cb22399b829f-cnibin\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:46.911748 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882492 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w5nn\" (UniqueName: \"kubernetes.io/projected/f4e9d179-2384-44e7-afd8-cb22399b829f-kube-api-access-7w5nn\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:46.911748 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882513 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret\") pod \"global-pull-secret-syncer-gbq5c\" (UID: \"ac3c6d61-29c8-4bc9-82d8-78177eb9045c\") " pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:46.911748 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882538 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5jlb\" (UniqueName: \"kubernetes.io/projected/35cfa5da-654d-4bb7-a263-b39acc0923bb-kube-api-access-w5jlb\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:46.911748 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882555 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-run-openvswitch\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.911748 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882574 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-host-run-multus-certs\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:46.911748 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882594 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f4e9d179-2384-44e7-afd8-cb22399b829f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:46.911748 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882672 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-sys\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:46.911748 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882700 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35cfa5da-654d-4bb7-a263-b39acc0923bb-tmp\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:46.911748 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882719 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-cni-bin\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.911748 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882743 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4e9d179-2384-44e7-afd8-cb22399b829f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:46.912377 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882761 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f4e9d179-2384-44e7-afd8-cb22399b829f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:46.912377 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882778 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-dbus\") pod \"global-pull-secret-syncer-gbq5c\" (UID: \"ac3c6d61-29c8-4bc9-82d8-78177eb9045c\") " pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:46.912377 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882817 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a8c9e0e-553f-4e86-af3a-0277d352166c-registration-dir\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:46.912377 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882845 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-run-ovn\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.912377 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882868 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-cni-binary-copy\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:46.912377 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882891 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-host-run-netns\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:46.912377 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882915 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs\") pod \"network-metrics-daemon-h6mc5\" (UID: \"a1fba745-34fb-4f41-b257-c6146b4f88fa\") " pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:38:46.912377 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882938 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a8c9e0e-553f-4e86-af3a-0277d352166c-device-dir\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:46.912377 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882961 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-host-var-lib-cni-multus\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:46.912377 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.882990 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q76x9\" (UniqueName: \"kubernetes.io/projected/747083d6-492d-413c-99e7-364aca1c1cee-kube-api-access-q76x9\") pod \"node-ca-rhlxp\" (UID: \"747083d6-492d-413c-99e7-364aca1c1cee\") " pod="openshift-image-registry/node-ca-rhlxp" Jul 08 17:38:46.912377 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883017 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e8468fb3-5642-43d6-95e5-c524fb532087-hosts-file\") pod \"node-resolver-8h4h6\" (UID: \"e8468fb3-5642-43d6-95e5-c524fb532087\") " pod="openshift-dns/node-resolver-8h4h6" Jul 08 17:38:46.912377 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883095 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e8468fb3-5642-43d6-95e5-c524fb532087-hosts-file\") pod \"node-resolver-8h4h6\" (UID: \"e8468fb3-5642-43d6-95e5-c524fb532087\") " pod="openshift-dns/node-resolver-8h4h6" Jul 08 17:38:46.912377 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883127 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvqtz\" (UniqueName: \"kubernetes.io/projected/a751057a-2392-4b54-86da-2fdb58461203-kube-api-access-vvqtz\") pod \"iptables-alerter-95k7k\" (UID: \"a751057a-2392-4b54-86da-2fdb58461203\") " pod="openshift-network-operator/iptables-alerter-95k7k" Jul 08 17:38:46.912377 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883156 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4e9d179-2384-44e7-afd8-cb22399b829f-cni-binary-copy\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:46.912377 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883178 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-kubelet\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.912377 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883207 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-run-netns\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.913081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883234 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-multus-daemon-config\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:46.913081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883267 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-host\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:46.913081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883292 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-log-socket\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.913081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883320 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-run-ovn-kubernetes\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.913081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883373 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b773719-d2d4-4be0-a119-71ffc944336c-ovnkube-config\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.913081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883388 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-host-var-lib-cni-bin\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:46.913081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883409 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klscm\" (UniqueName: \"kubernetes.io/projected/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-kube-api-access-klscm\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:46.913081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883447 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a751057a-2392-4b54-86da-2fdb58461203-iptables-alerter-script\") pod \"iptables-alerter-95k7k\" (UID: \"a751057a-2392-4b54-86da-2fdb58461203\") " pod="openshift-network-operator/iptables-alerter-95k7k" Jul 08 17:38:46.913081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883467 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-etc-openvswitch\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.913081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883481 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f4e9d179-2384-44e7-afd8-cb22399b829f-os-release\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:46.913081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883503 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-kubelet-config\") pod \"global-pull-secret-syncer-gbq5c\" (UID: \"ac3c6d61-29c8-4bc9-82d8-78177eb9045c\") " pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:46.913081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883517 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-sysctl-d\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:46.913081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883532 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-system-cni-dir\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:46.913081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883555 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-host-run-k8s-cni-cncf-io\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:46.913081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883570 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g7r7\" (UniqueName: \"kubernetes.io/projected/a1fba745-34fb-4f41-b257-c6146b4f88fa-kube-api-access-9g7r7\") pod \"network-metrics-daemon-h6mc5\" (UID: \"a1fba745-34fb-4f41-b257-c6146b4f88fa\") " pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:38:46.913081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883588 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-systemd-units\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.913813 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883610 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-run-systemd\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.913813 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883631 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/747083d6-492d-413c-99e7-364aca1c1cee-host\") pod \"node-ca-rhlxp\" (UID: \"747083d6-492d-413c-99e7-364aca1c1cee\") " pod="openshift-image-registry/node-ca-rhlxp" Jul 08 17:38:46.913813 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883648 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-kubernetes\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:46.913813 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883676 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-run\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:46.913813 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883684 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/747083d6-492d-413c-99e7-364aca1c1cee-host\") pod \"node-ca-rhlxp\" (UID: \"747083d6-492d-413c-99e7-364aca1c1cee\") " pod="openshift-image-registry/node-ca-rhlxp" Jul 08 17:38:46.913813 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883703 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a8c9e0e-553f-4e86-af3a-0277d352166c-socket-dir\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:46.913813 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883731 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w89bv\" (UniqueName: \"kubernetes.io/projected/3a8c9e0e-553f-4e86-af3a-0277d352166c-kube-api-access-w89bv\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:46.913813 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883747 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2p7m\" (UniqueName: \"kubernetes.io/projected/3b773719-d2d4-4be0-a119-71ffc944336c-kube-api-access-g2p7m\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.913813 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883777 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jqcn\" (UniqueName: \"kubernetes.io/projected/e8468fb3-5642-43d6-95e5-c524fb532087-kube-api-access-7jqcn\") pod \"node-resolver-8h4h6\" (UID: \"e8468fb3-5642-43d6-95e5-c524fb532087\") " pod="openshift-dns/node-resolver-8h4h6" Jul 08 17:38:46.913813 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883803 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-tuned\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:46.913813 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883818 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-var-lib-openvswitch\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.913813 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883837 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-cni-netd\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.913813 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883930 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/970dfcb3-0c92-4a96-95c9-fe1db4c711d6-konnectivity-ca\") pod \"konnectivity-agent-bl792\" (UID: \"970dfcb3-0c92-4a96-95c9-fe1db4c711d6\") " pod="kube-system/konnectivity-agent-bl792" Jul 08 17:38:46.913813 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883947 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-os-release\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:46.913813 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883962 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-etc-kubernetes\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:46.913813 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.883985 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4ksh\" (UniqueName: \"kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh\") pod \"network-check-target-d6xks\" (UID: \"135de3f6-fc59-4070-9899-88ac1d14ffde\") " pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:38:46.913813 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884014 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/747083d6-492d-413c-99e7-364aca1c1cee-serviceca\") pod \"node-ca-rhlxp\" (UID: \"747083d6-492d-413c-99e7-364aca1c1cee\") " pod="openshift-image-registry/node-ca-rhlxp" Jul 08 17:38:46.914490 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884144 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8468fb3-5642-43d6-95e5-c524fb532087-tmp-dir\") pod \"node-resolver-8h4h6\" (UID: \"e8468fb3-5642-43d6-95e5-c524fb532087\") " pod="openshift-dns/node-resolver-8h4h6" Jul 08 17:38:46.914490 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884252 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-systemd\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:46.914490 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884295 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-lib-modules\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:46.914490 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884318 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/970dfcb3-0c92-4a96-95c9-fe1db4c711d6-agent-certs\") pod \"konnectivity-agent-bl792\" (UID: \"970dfcb3-0c92-4a96-95c9-fe1db4c711d6\") " pod="kube-system/konnectivity-agent-bl792" Jul 08 17:38:46.914490 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884346 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-cnibin\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:46.914490 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884372 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-multus-socket-dir-parent\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:46.914490 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884399 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-host-var-lib-kubelet\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:46.914490 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884447 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4e9d179-2384-44e7-afd8-cb22399b829f-system-cni-dir\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:46.914490 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884479 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-modprobe-d\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:46.914490 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884502 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8468fb3-5642-43d6-95e5-c524fb532087-tmp-dir\") pod \"node-resolver-8h4h6\" (UID: \"e8468fb3-5642-43d6-95e5-c524fb532087\") " pod="openshift-dns/node-resolver-8h4h6" Jul 08 17:38:46.914490 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884508 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a8c9e0e-553f-4e86-af3a-0277d352166c-etc-selinux\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:46.914490 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884579 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/747083d6-492d-413c-99e7-364aca1c1cee-serviceca\") pod \"node-ca-rhlxp\" (UID: \"747083d6-492d-413c-99e7-364aca1c1cee\") " pod="openshift-image-registry/node-ca-rhlxp" Jul 08 17:38:46.914490 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884593 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/970dfcb3-0c92-4a96-95c9-fe1db4c711d6-konnectivity-ca\") pod \"konnectivity-agent-bl792\" (UID: \"970dfcb3-0c92-4a96-95c9-fe1db4c711d6\") " pod="kube-system/konnectivity-agent-bl792" Jul 08 17:38:46.914490 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884601 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-slash\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.914490 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884623 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-hostroot\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:46.914490 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884639 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a751057a-2392-4b54-86da-2fdb58461203-host-slash\") pod \"iptables-alerter-95k7k\" (UID: \"a751057a-2392-4b54-86da-2fdb58461203\") " pod="openshift-network-operator/iptables-alerter-95k7k" Jul 08 17:38:46.914490 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884657 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-var-lib-kubelet\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:46.915354 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884679 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a8c9e0e-553f-4e86-af3a-0277d352166c-sys-fs\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:46.915354 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884706 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-node-log\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.915354 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884693 2584 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 08 17:38:46.915354 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884726 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.915354 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.884742 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b773719-d2d4-4be0-a119-71ffc944336c-env-overrides\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:46.915354 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.888696 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/970dfcb3-0c92-4a96-95c9-fe1db4c711d6-agent-certs\") pod \"konnectivity-agent-bl792\" (UID: \"970dfcb3-0c92-4a96-95c9-fe1db4c711d6\") " pod="kube-system/konnectivity-agent-bl792" Jul 08 17:38:46.915354 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:46.890983 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:38:46.915354 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:46.891002 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:38:46.915354 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:46.891015 2584 projected.go:196] Error preparing data for projected volume kube-api-access-k4ksh for pod openshift-network-diagnostics/network-check-target-d6xks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:38:46.915354 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:46.891088 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh podName:135de3f6-fc59-4070-9899-88ac1d14ffde nodeName:}" failed. No retries permitted until 2026-07-08 17:38:47.391065807 +0000 UTC m=+3.075163087 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k4ksh" (UniqueName: "kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh") pod "network-check-target-d6xks" (UID: "135de3f6-fc59-4070-9899-88ac1d14ffde") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:38:46.915354 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.893339 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q76x9\" (UniqueName: \"kubernetes.io/projected/747083d6-492d-413c-99e7-364aca1c1cee-kube-api-access-q76x9\") pod \"node-ca-rhlxp\" (UID: \"747083d6-492d-413c-99e7-364aca1c1cee\") " pod="openshift-image-registry/node-ca-rhlxp" Jul 08 17:38:46.915354 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.893358 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jqcn\" (UniqueName: \"kubernetes.io/projected/e8468fb3-5642-43d6-95e5-c524fb532087-kube-api-access-7jqcn\") pod \"node-resolver-8h4h6\" (UID: \"e8468fb3-5642-43d6-95e5-c524fb532087\") " pod="openshift-dns/node-resolver-8h4h6" Jul 08 17:38:47.042028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.954310 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal" event={"ID":"66beb382798cae7e66e45d75b5f0b9d7","Type":"ContainerStarted","Data":"1c935fe7eea87113a6c5e94961ce9ee9f02d6d68db6efdda6482fbbb6a6d36c3"} Jul 08 17:38:47.042028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.955349 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal" event={"ID":"8119fcfc2e17d9ba7073558077e669dc","Type":"ContainerStarted","Data":"2ddc5f50edd2386a670db1996d9a0aa446e292f8643a9b456ad8533651358184"} Jul 08 17:38:47.042028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.985912 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-etc-kubernetes\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.042028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.985960 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-systemd\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.042028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986111 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-lib-modules\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.042028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986157 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-etc-kubernetes\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.042028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986181 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-lib-modules\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.042028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986288 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-systemd\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.042028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986314 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-cnibin\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.042028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986335 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-multus-socket-dir-parent\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.042028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986374 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-host-var-lib-kubelet\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.042028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986405 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4e9d179-2384-44e7-afd8-cb22399b829f-system-cni-dir\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:47.042028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986428 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-modprobe-d\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.042028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986441 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-multus-socket-dir-parent\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.042028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986449 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a8c9e0e-553f-4e86-af3a-0277d352166c-etc-selinux\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:47.042028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986483 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-cnibin\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.042028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986500 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3a8c9e0e-553f-4e86-af3a-0277d352166c-etc-selinux\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:47.042028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986505 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-slash\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.043200 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986558 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-slash\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.043200 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986586 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-host-var-lib-kubelet\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.043200 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986610 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-modprobe-d\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.043200 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986614 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-hostroot\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.043200 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986526 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-hostroot\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.043200 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986651 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a751057a-2392-4b54-86da-2fdb58461203-host-slash\") pod \"iptables-alerter-95k7k\" (UID: \"a751057a-2392-4b54-86da-2fdb58461203\") " pod="openshift-network-operator/iptables-alerter-95k7k" Jul 08 17:38:47.043200 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986656 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4e9d179-2384-44e7-afd8-cb22399b829f-system-cni-dir\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:47.043200 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986675 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-var-lib-kubelet\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.043200 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986697 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a751057a-2392-4b54-86da-2fdb58461203-host-slash\") pod \"iptables-alerter-95k7k\" (UID: \"a751057a-2392-4b54-86da-2fdb58461203\") " pod="openshift-network-operator/iptables-alerter-95k7k" Jul 08 17:38:47.043200 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986701 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a8c9e0e-553f-4e86-af3a-0277d352166c-sys-fs\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:47.043200 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986733 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-node-log\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.043200 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986756 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.043200 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986779 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b773719-d2d4-4be0-a119-71ffc944336c-env-overrides\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.043200 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986789 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.043200 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986799 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b773719-d2d4-4be0-a119-71ffc944336c-ovnkube-script-lib\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.043200 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986816 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-sysconfig\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.043200 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986829 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-node-log\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.045074 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986835 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a8c9e0e-553f-4e86-af3a-0277d352166c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:47.045074 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986861 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b773719-d2d4-4be0-a119-71ffc944336c-ovn-node-metrics-cert\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.045074 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986867 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3a8c9e0e-553f-4e86-af3a-0277d352166c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:47.045074 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986758 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3a8c9e0e-553f-4e86-af3a-0277d352166c-sys-fs\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:47.045074 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986903 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-sysctl-conf\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.045074 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986927 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-multus-conf-dir\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.045074 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986947 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-multus-cni-dir\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.045074 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986969 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4e9d179-2384-44e7-afd8-cb22399b829f-cnibin\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:47.045074 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.986991 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5nn\" (UniqueName: \"kubernetes.io/projected/f4e9d179-2384-44e7-afd8-cb22399b829f-kube-api-access-7w5nn\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:47.045074 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.987008 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret\") pod \"global-pull-secret-syncer-gbq5c\" (UID: \"ac3c6d61-29c8-4bc9-82d8-78177eb9045c\") " pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:47.045074 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.987034 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5jlb\" (UniqueName: \"kubernetes.io/projected/35cfa5da-654d-4bb7-a263-b39acc0923bb-kube-api-access-w5jlb\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.045074 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.987054 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-run-openvswitch\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.045074 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.987093 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-run-openvswitch\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.045074 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.987124 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-multus-conf-dir\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.045074 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.987219 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-multus-cni-dir\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.045074 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:46.987506 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:38:47.045074 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.987522 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-sysctl-conf\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.045865 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:46.987565 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret podName:ac3c6d61-29c8-4bc9-82d8-78177eb9045c nodeName:}" failed. No retries permitted until 2026-07-08 17:38:47.487551285 +0000 UTC m=+3.171648562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret") pod "global-pull-secret-syncer-gbq5c" (UID: "ac3c6d61-29c8-4bc9-82d8-78177eb9045c") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:38:47.045865 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.987705 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-host-run-multus-certs\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.045865 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.987733 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f4e9d179-2384-44e7-afd8-cb22399b829f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:47.045865 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.987753 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-sys\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.045865 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.987777 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35cfa5da-654d-4bb7-a263-b39acc0923bb-tmp\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.045865 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.987797 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-cni-bin\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.045865 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.987819 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4e9d179-2384-44e7-afd8-cb22399b829f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:47.045865 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.987842 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f4e9d179-2384-44e7-afd8-cb22399b829f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:47.045865 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.987862 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-dbus\") pod \"global-pull-secret-syncer-gbq5c\" (UID: \"ac3c6d61-29c8-4bc9-82d8-78177eb9045c\") " pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:47.045865 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.987881 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a8c9e0e-553f-4e86-af3a-0277d352166c-registration-dir\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:47.045865 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.987901 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-run-ovn\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.045865 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.987920 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-cni-binary-copy\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.045865 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.987939 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-host-run-netns\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.045865 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.988006 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-sysconfig\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.045865 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.988021 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-sys\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.045865 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.988072 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-var-lib-kubelet\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.045865 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:46.988077 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:38:47.046665 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:46.988113 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs podName:a1fba745-34fb-4f41-b257-c6146b4f88fa nodeName:}" failed. No retries permitted until 2026-07-08 17:38:47.488098462 +0000 UTC m=+3.172195718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs") pod "network-metrics-daemon-h6mc5" (UID: "a1fba745-34fb-4f41-b257-c6146b4f88fa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:38:47.046665 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.988174 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4e9d179-2384-44e7-afd8-cb22399b829f-cnibin\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:47.046665 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.988526 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f4e9d179-2384-44e7-afd8-cb22399b829f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:47.046665 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.988557 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs\") pod \"network-metrics-daemon-h6mc5\" (UID: \"a1fba745-34fb-4f41-b257-c6146b4f88fa\") " pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:38:47.046665 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.988568 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3b773719-d2d4-4be0-a119-71ffc944336c-env-overrides\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.046665 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.988788 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3b773719-d2d4-4be0-a119-71ffc944336c-ovnkube-script-lib\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.046665 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.988929 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-cni-bin\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.046665 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.989051 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4e9d179-2384-44e7-afd8-cb22399b829f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:47.046665 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.989342 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b773719-d2d4-4be0-a119-71ffc944336c-ovn-node-metrics-cert\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.046665 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.989366 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-dbus\") pod \"global-pull-secret-syncer-gbq5c\" (UID: \"ac3c6d61-29c8-4bc9-82d8-78177eb9045c\") " pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:47.046665 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.989411 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3a8c9e0e-553f-4e86-af3a-0277d352166c-registration-dir\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:47.046665 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.989460 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-run-ovn\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.046665 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.989587 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f4e9d179-2384-44e7-afd8-cb22399b829f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:47.046665 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.989680 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-host-run-netns\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.046665 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.989736 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-host-run-multus-certs\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.046665 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.989815 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a8c9e0e-553f-4e86-af3a-0277d352166c-device-dir\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:47.047373 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.989844 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-host-var-lib-cni-multus\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.047373 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.989849 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-cni-binary-copy\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.047373 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.989878 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvqtz\" (UniqueName: \"kubernetes.io/projected/a751057a-2392-4b54-86da-2fdb58461203-kube-api-access-vvqtz\") pod \"iptables-alerter-95k7k\" (UID: \"a751057a-2392-4b54-86da-2fdb58461203\") " pod="openshift-network-operator/iptables-alerter-95k7k" Jul 08 17:38:47.047373 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.989920 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4e9d179-2384-44e7-afd8-cb22399b829f-cni-binary-copy\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:47.047373 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.989969 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-kubelet\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.047373 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.989991 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-kubelet\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.047373 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.989995 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-run-netns\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.047373 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.990025 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-multus-daemon-config\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.047373 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.990054 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-host\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.047373 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.990084 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-log-socket\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.047373 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.990116 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-run-ovn-kubernetes\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.047373 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.990025 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-run-netns\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.047373 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.990190 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-host\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.047373 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.990243 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-log-socket\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.047373 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.990485 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35cfa5da-654d-4bb7-a263-b39acc0923bb-tmp\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.047373 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.990504 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4e9d179-2384-44e7-afd8-cb22399b829f-cni-binary-copy\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:47.047373 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.990566 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3a8c9e0e-553f-4e86-af3a-0277d352166c-device-dir\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:47.048141 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.990631 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-run-ovn-kubernetes\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.048141 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.990602 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-host-var-lib-cni-multus\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.048141 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.990962 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b773719-d2d4-4be0-a119-71ffc944336c-ovnkube-config\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.048141 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.990992 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-host-var-lib-cni-bin\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.048141 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991010 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klscm\" (UniqueName: \"kubernetes.io/projected/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-kube-api-access-klscm\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.048141 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991029 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-multus-daemon-config\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.048141 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991031 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a751057a-2392-4b54-86da-2fdb58461203-iptables-alerter-script\") pod \"iptables-alerter-95k7k\" (UID: \"a751057a-2392-4b54-86da-2fdb58461203\") " pod="openshift-network-operator/iptables-alerter-95k7k" Jul 08 17:38:47.048141 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991047 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3b773719-d2d4-4be0-a119-71ffc944336c-ovnkube-config\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.048141 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991079 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-etc-openvswitch\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.048141 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991108 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f4e9d179-2384-44e7-afd8-cb22399b829f-os-release\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:47.048141 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991111 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-host-var-lib-cni-bin\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.048141 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991149 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-kubelet-config\") pod \"global-pull-secret-syncer-gbq5c\" (UID: \"ac3c6d61-29c8-4bc9-82d8-78177eb9045c\") " pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:47.048141 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991164 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-kubelet-config\") pod \"global-pull-secret-syncer-gbq5c\" (UID: \"ac3c6d61-29c8-4bc9-82d8-78177eb9045c\") " pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:47.048141 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991177 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-sysctl-d\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.048141 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991207 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-etc-openvswitch\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.048141 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991207 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-system-cni-dir\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.048141 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991241 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-host-run-k8s-cni-cncf-io\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.048863 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991275 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g7r7\" (UniqueName: \"kubernetes.io/projected/a1fba745-34fb-4f41-b257-c6146b4f88fa-kube-api-access-9g7r7\") pod \"network-metrics-daemon-h6mc5\" (UID: \"a1fba745-34fb-4f41-b257-c6146b4f88fa\") " pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:38:47.048863 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991295 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-host-run-k8s-cni-cncf-io\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.048863 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991314 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-systemd-units\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.048863 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991256 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f4e9d179-2384-44e7-afd8-cb22399b829f-os-release\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:47.048863 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991365 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-system-cni-dir\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.048863 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991404 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a751057a-2392-4b54-86da-2fdb58461203-iptables-alerter-script\") pod \"iptables-alerter-95k7k\" (UID: \"a751057a-2392-4b54-86da-2fdb58461203\") " pod="openshift-network-operator/iptables-alerter-95k7k" Jul 08 17:38:47.048863 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991510 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-run-systemd\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.048863 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991528 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-sysctl-d\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.048863 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991535 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-kubernetes\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.048863 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991566 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-run\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.048863 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991569 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-systemd-units\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.048863 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991607 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-run\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.048863 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991720 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a8c9e0e-553f-4e86-af3a-0277d352166c-socket-dir\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:47.048863 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991756 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w89bv\" (UniqueName: \"kubernetes.io/projected/3a8c9e0e-553f-4e86-af3a-0277d352166c-kube-api-access-w89bv\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:47.048863 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991813 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2p7m\" (UniqueName: \"kubernetes.io/projected/3b773719-d2d4-4be0-a119-71ffc944336c-kube-api-access-g2p7m\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.048863 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991842 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-tuned\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.048863 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991869 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-var-lib-openvswitch\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.049733 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991889 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-cni-netd\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.049733 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991894 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3a8c9e0e-553f-4e86-af3a-0277d352166c-socket-dir\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:47.049733 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991912 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-os-release\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.049733 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991937 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-run-systemd\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.049733 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.991984 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-kubernetes\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.049733 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.992158 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-var-lib-openvswitch\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.049733 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.992209 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3b773719-d2d4-4be0-a119-71ffc944336c-host-cni-netd\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.049733 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.992207 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-os-release\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.049733 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:46.993852 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/35cfa5da-654d-4bb7-a263-b39acc0923bb-etc-tuned\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.049733 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.003034 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5nn\" (UniqueName: \"kubernetes.io/projected/f4e9d179-2384-44e7-afd8-cb22399b829f-kube-api-access-7w5nn\") pod \"multus-additional-cni-plugins-blvkt\" (UID: \"f4e9d179-2384-44e7-afd8-cb22399b829f\") " pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:47.049733 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.011223 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvqtz\" (UniqueName: \"kubernetes.io/projected/a751057a-2392-4b54-86da-2fdb58461203-kube-api-access-vvqtz\") pod \"iptables-alerter-95k7k\" (UID: \"a751057a-2392-4b54-86da-2fdb58461203\") " pod="openshift-network-operator/iptables-alerter-95k7k" Jul 08 17:38:47.049733 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.011386 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w89bv\" (UniqueName: \"kubernetes.io/projected/3a8c9e0e-553f-4e86-af3a-0277d352166c-kube-api-access-w89bv\") pod \"aws-ebs-csi-driver-node-49bpp\" (UID: \"3a8c9e0e-553f-4e86-af3a-0277d352166c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:47.049733 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.011623 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klscm\" (UniqueName: \"kubernetes.io/projected/67ae88d7-7fa6-494a-9bd0-43c15921cb8c-kube-api-access-klscm\") pod \"multus-8zt5s\" (UID: \"67ae88d7-7fa6-494a-9bd0-43c15921cb8c\") " pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.049733 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.011726 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5jlb\" (UniqueName: \"kubernetes.io/projected/35cfa5da-654d-4bb7-a263-b39acc0923bb-kube-api-access-w5jlb\") pod \"tuned-4zlvf\" (UID: \"35cfa5da-654d-4bb7-a263-b39acc0923bb\") " pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.049733 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.011879 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2p7m\" (UniqueName: \"kubernetes.io/projected/3b773719-d2d4-4be0-a119-71ffc944336c-kube-api-access-g2p7m\") pod \"ovnkube-node-krx82\" (UID: \"3b773719-d2d4-4be0-a119-71ffc944336c\") " pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.049733 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.012010 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g7r7\" (UniqueName: \"kubernetes.io/projected/a1fba745-34fb-4f41-b257-c6146b4f88fa-kube-api-access-9g7r7\") pod \"network-metrics-daemon-h6mc5\" (UID: \"a1fba745-34fb-4f41-b257-c6146b4f88fa\") " pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:38:47.058278 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.058242 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bl792" Jul 08 17:38:47.064374 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.064358 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rhlxp" Jul 08 17:38:47.069468 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:47.069439 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod970dfcb3_0c92_4a96_95c9_fe1db4c711d6.slice/crio-2d6399df43db204027be808b8b12188f3d9ab582238fc2030f2eb37901367aa9 WatchSource:0}: Error finding container 2d6399df43db204027be808b8b12188f3d9ab582238fc2030f2eb37901367aa9: Status 404 returned error can't find the container with id 2d6399df43db204027be808b8b12188f3d9ab582238fc2030f2eb37901367aa9 Jul 08 17:38:47.072906 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.072887 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8h4h6" Jul 08 17:38:47.073734 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:47.073712 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod747083d6_492d_413c_99e7_364aca1c1cee.slice/crio-cb1a002297bc460a4becb8938560f16d19a3eaef50a18c0abdd2ba8cbd417763 WatchSource:0}: Error finding container cb1a002297bc460a4becb8938560f16d19a3eaef50a18c0abdd2ba8cbd417763: Status 404 returned error can't find the container with id cb1a002297bc460a4becb8938560f16d19a3eaef50a18c0abdd2ba8cbd417763 Jul 08 17:38:47.076666 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.076647 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8zt5s" Jul 08 17:38:47.083839 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:47.083813 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8468fb3_5642_43d6_95e5_c524fb532087.slice/crio-0de8e93770d0211cfce0557b81f59eaddbe880fe5ec452f581ffe502a4bb3685 WatchSource:0}: Error finding container 0de8e93770d0211cfce0557b81f59eaddbe880fe5ec452f581ffe502a4bb3685: Status 404 returned error can't find the container with id 0de8e93770d0211cfce0557b81f59eaddbe880fe5ec452f581ffe502a4bb3685 Jul 08 17:38:47.085790 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.085769 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-blvkt" Jul 08 17:38:47.087570 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:47.087550 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ae88d7_7fa6_494a_9bd0_43c15921cb8c.slice/crio-80ee06f60a17463245233695736961ccfece60c24c3ec63321f26a7bb6726d5d WatchSource:0}: Error finding container 80ee06f60a17463245233695736961ccfece60c24c3ec63321f26a7bb6726d5d: Status 404 returned error can't find the container with id 80ee06f60a17463245233695736961ccfece60c24c3ec63321f26a7bb6726d5d Jul 08 17:38:47.094194 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.094178 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" Jul 08 17:38:47.095829 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:47.095796 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e9d179_2384_44e7_afd8_cb22399b829f.slice/crio-f721fd04dc8addd138ef2301596c76c21f1f3cacaf50d0748e913050b1789f7e WatchSource:0}: Error finding container f721fd04dc8addd138ef2301596c76c21f1f3cacaf50d0748e913050b1789f7e: Status 404 returned error can't find the container with id f721fd04dc8addd138ef2301596c76c21f1f3cacaf50d0748e913050b1789f7e Jul 08 17:38:47.099132 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.099114 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" Jul 08 17:38:47.103016 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:47.102987 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cfa5da_654d_4bb7_a263_b39acc0923bb.slice/crio-8d93a016cf881cef15f594233bb0d8df9bccabc24cc2ac9ceee4b04ce8c33fef WatchSource:0}: Error finding container 8d93a016cf881cef15f594233bb0d8df9bccabc24cc2ac9ceee4b04ce8c33fef: Status 404 returned error can't find the container with id 8d93a016cf881cef15f594233bb0d8df9bccabc24cc2ac9ceee4b04ce8c33fef Jul 08 17:38:47.108542 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:47.108517 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8c9e0e_553f_4e86_af3a_0277d352166c.slice/crio-0424c20db0a09bb68ec3496408ab1a1b71e9aca68ef03e7490db3e5b9691b7eb WatchSource:0}: Error finding container 0424c20db0a09bb68ec3496408ab1a1b71e9aca68ef03e7490db3e5b9691b7eb: Status 404 returned error can't find the container with id 0424c20db0a09bb68ec3496408ab1a1b71e9aca68ef03e7490db3e5b9691b7eb Jul 08 17:38:47.109572 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.109556 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-95k7k" Jul 08 17:38:47.114577 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.114563 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:38:47.119294 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:47.119277 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda751057a_2392_4b54_86da_2fdb58461203.slice/crio-3873e062c3809dac51845dfa804938e4b66bedbb0a51ba082b547ccd10ab8806 WatchSource:0}: Error finding container 3873e062c3809dac51845dfa804938e4b66bedbb0a51ba082b547ccd10ab8806: Status 404 returned error can't find the container with id 3873e062c3809dac51845dfa804938e4b66bedbb0a51ba082b547ccd10ab8806 Jul 08 17:38:47.123781 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:38:47.123755 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b773719_d2d4_4be0_a119_71ffc944336c.slice/crio-ab6fdc55a8f166088be1dba22146e58ce626dbf0b960af38452d58763d54fa55 WatchSource:0}: Error finding container ab6fdc55a8f166088be1dba22146e58ce626dbf0b960af38452d58763d54fa55: Status 404 returned error can't find the container with id ab6fdc55a8f166088be1dba22146e58ce626dbf0b960af38452d58763d54fa55 Jul 08 17:38:47.259744 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.259670 2584 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 17:38:47.396214 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.395717 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4ksh\" (UniqueName: \"kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh\") pod \"network-check-target-d6xks\" (UID: \"135de3f6-fc59-4070-9899-88ac1d14ffde\") " pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:38:47.396214 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:47.395943 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:38:47.396214 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:47.395969 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:38:47.396214 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:47.395983 2584 projected.go:196] Error preparing data for projected volume kube-api-access-k4ksh for pod openshift-network-diagnostics/network-check-target-d6xks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:38:47.396214 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:47.396048 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh podName:135de3f6-fc59-4070-9899-88ac1d14ffde nodeName:}" failed. No retries permitted until 2026-07-08 17:38:48.396028983 +0000 UTC m=+4.080126246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4ksh" (UniqueName: "kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh") pod "network-check-target-d6xks" (UID: "135de3f6-fc59-4070-9899-88ac1d14ffde") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:38:47.496849 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.496809 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret\") pod \"global-pull-secret-syncer-gbq5c\" (UID: \"ac3c6d61-29c8-4bc9-82d8-78177eb9045c\") " pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:47.497060 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.496867 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs\") pod \"network-metrics-daemon-h6mc5\" (UID: \"a1fba745-34fb-4f41-b257-c6146b4f88fa\") " pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:38:47.497060 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:47.496994 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:38:47.497202 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:47.497066 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs podName:a1fba745-34fb-4f41-b257-c6146b4f88fa nodeName:}" failed. No retries permitted until 2026-07-08 17:38:48.497047185 +0000 UTC m=+4.181144457 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs") pod "network-metrics-daemon-h6mc5" (UID: "a1fba745-34fb-4f41-b257-c6146b4f88fa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:38:47.497202 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:47.496994 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:38:47.497202 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:47.497158 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret podName:ac3c6d61-29c8-4bc9-82d8-78177eb9045c nodeName:}" failed. No retries permitted until 2026-07-08 17:38:48.497132312 +0000 UTC m=+4.181229584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret") pod "global-pull-secret-syncer-gbq5c" (UID: "ac3c6d61-29c8-4bc9-82d8-78177eb9045c") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:38:47.801603 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.801557 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 17:33:46 +0000 UTC" deadline="2028-02-13 17:19:30.758011607 +0000 UTC" Jul 08 17:38:47.801603 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.801600 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14039h40m42.956416166s" Jul 08 17:38:47.950520 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.950487 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:38:47.950989 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:47.950632 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:38:47.951046 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.951021 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:38:47.951129 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:47.951110 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:38:47.951197 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.951174 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:47.951291 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:47.951262 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:38:47.960787 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.960755 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8h4h6" event={"ID":"e8468fb3-5642-43d6-95e5-c524fb532087","Type":"ContainerStarted","Data":"0de8e93770d0211cfce0557b81f59eaddbe880fe5ec452f581ffe502a4bb3685"} Jul 08 17:38:47.962344 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.962318 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rhlxp" event={"ID":"747083d6-492d-413c-99e7-364aca1c1cee","Type":"ContainerStarted","Data":"cb1a002297bc460a4becb8938560f16d19a3eaef50a18c0abdd2ba8cbd417763"} Jul 08 17:38:47.964162 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.964127 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bl792" event={"ID":"970dfcb3-0c92-4a96-95c9-fe1db4c711d6","Type":"ContainerStarted","Data":"2d6399df43db204027be808b8b12188f3d9ab582238fc2030f2eb37901367aa9"} Jul 08 17:38:47.966033 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.966007 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blvkt" event={"ID":"f4e9d179-2384-44e7-afd8-cb22399b829f","Type":"ContainerStarted","Data":"f721fd04dc8addd138ef2301596c76c21f1f3cacaf50d0748e913050b1789f7e"} Jul 08 17:38:47.968214 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.968191 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-krx82" event={"ID":"3b773719-d2d4-4be0-a119-71ffc944336c","Type":"ContainerStarted","Data":"ab6fdc55a8f166088be1dba22146e58ce626dbf0b960af38452d58763d54fa55"} Jul 08 17:38:47.971721 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.971698 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-95k7k" event={"ID":"a751057a-2392-4b54-86da-2fdb58461203","Type":"ContainerStarted","Data":"3873e062c3809dac51845dfa804938e4b66bedbb0a51ba082b547ccd10ab8806"} Jul 08 17:38:47.975833 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.975806 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" event={"ID":"3a8c9e0e-553f-4e86-af3a-0277d352166c","Type":"ContainerStarted","Data":"0424c20db0a09bb68ec3496408ab1a1b71e9aca68ef03e7490db3e5b9691b7eb"} Jul 08 17:38:47.979264 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.979228 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" event={"ID":"35cfa5da-654d-4bb7-a263-b39acc0923bb","Type":"ContainerStarted","Data":"8d93a016cf881cef15f594233bb0d8df9bccabc24cc2ac9ceee4b04ce8c33fef"} Jul 08 17:38:47.982315 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:47.982235 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8zt5s" event={"ID":"67ae88d7-7fa6-494a-9bd0-43c15921cb8c","Type":"ContainerStarted","Data":"80ee06f60a17463245233695736961ccfece60c24c3ec63321f26a7bb6726d5d"} Jul 08 17:38:48.404711 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:48.404666 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4ksh\" (UniqueName: \"kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh\") pod \"network-check-target-d6xks\" (UID: \"135de3f6-fc59-4070-9899-88ac1d14ffde\") " pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:38:48.404946 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:48.404899 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:38:48.404946 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:48.404925 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:38:48.404946 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:48.404939 2584 projected.go:196] Error preparing data for projected volume kube-api-access-k4ksh for pod openshift-network-diagnostics/network-check-target-d6xks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:38:48.405143 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:48.405000 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh podName:135de3f6-fc59-4070-9899-88ac1d14ffde nodeName:}" failed. No retries permitted until 2026-07-08 17:38:50.404982128 +0000 UTC m=+6.089079386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4ksh" (UniqueName: "kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh") pod "network-check-target-d6xks" (UID: "135de3f6-fc59-4070-9899-88ac1d14ffde") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:38:48.505466 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:48.505399 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret\") pod \"global-pull-secret-syncer-gbq5c\" (UID: \"ac3c6d61-29c8-4bc9-82d8-78177eb9045c\") " pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:48.505466 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:48.505460 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs\") pod \"network-metrics-daemon-h6mc5\" (UID: \"a1fba745-34fb-4f41-b257-c6146b4f88fa\") " pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:38:48.505703 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:48.505591 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:38:48.505703 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:48.505656 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs podName:a1fba745-34fb-4f41-b257-c6146b4f88fa nodeName:}" failed. No retries permitted until 2026-07-08 17:38:50.505635408 +0000 UTC m=+6.189732676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs") pod "network-metrics-daemon-h6mc5" (UID: "a1fba745-34fb-4f41-b257-c6146b4f88fa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:38:48.506074 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:48.506055 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:38:48.506153 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:48.506110 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret podName:ac3c6d61-29c8-4bc9-82d8-78177eb9045c nodeName:}" failed. No retries permitted until 2026-07-08 17:38:50.506094069 +0000 UTC m=+6.190191324 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret") pod "global-pull-secret-syncer-gbq5c" (UID: "ac3c6d61-29c8-4bc9-82d8-78177eb9045c") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:38:48.802389 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:48.802287 2584 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 17:33:46 +0000 UTC" deadline="2028-02-21 20:52:59.268899601 +0000 UTC" Jul 08 17:38:48.802389 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:48.802329 2584 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14235h14m10.466575601s" Jul 08 17:38:49.951309 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:49.950426 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:49.951309 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:49.950558 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:38:49.951309 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:49.950624 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:38:49.951309 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:49.950708 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:38:49.951309 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:49.950748 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:38:49.951309 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:49.950821 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:38:50.422997 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:50.422895 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4ksh\" (UniqueName: \"kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh\") pod \"network-check-target-d6xks\" (UID: \"135de3f6-fc59-4070-9899-88ac1d14ffde\") " pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:38:50.423161 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:50.423096 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:38:50.423161 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:50.423125 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:38:50.423161 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:50.423137 2584 projected.go:196] Error preparing data for projected volume kube-api-access-k4ksh for pod openshift-network-diagnostics/network-check-target-d6xks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:38:50.423380 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:50.423194 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh podName:135de3f6-fc59-4070-9899-88ac1d14ffde nodeName:}" failed. No retries permitted until 2026-07-08 17:38:54.423174962 +0000 UTC m=+10.107272243 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4ksh" (UniqueName: "kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh") pod "network-check-target-d6xks" (UID: "135de3f6-fc59-4070-9899-88ac1d14ffde") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:38:50.524119 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:50.524085 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret\") pod \"global-pull-secret-syncer-gbq5c\" (UID: \"ac3c6d61-29c8-4bc9-82d8-78177eb9045c\") " pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:50.524541 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:50.524137 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs\") pod \"network-metrics-daemon-h6mc5\" (UID: \"a1fba745-34fb-4f41-b257-c6146b4f88fa\") " pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:38:50.524541 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:50.524298 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:38:50.524541 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:50.524361 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs podName:a1fba745-34fb-4f41-b257-c6146b4f88fa nodeName:}" failed. No retries permitted until 2026-07-08 17:38:54.524343113 +0000 UTC m=+10.208440371 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs") pod "network-metrics-daemon-h6mc5" (UID: "a1fba745-34fb-4f41-b257-c6146b4f88fa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:38:50.524541 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:50.524364 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:38:50.524541 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:50.524422 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret podName:ac3c6d61-29c8-4bc9-82d8-78177eb9045c nodeName:}" failed. No retries permitted until 2026-07-08 17:38:54.524406974 +0000 UTC m=+10.208504251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret") pod "global-pull-secret-syncer-gbq5c" (UID: "ac3c6d61-29c8-4bc9-82d8-78177eb9045c") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:38:51.950733 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:51.950695 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:38:51.951199 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:51.950842 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:38:51.951287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:51.951262 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:38:51.951546 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:51.951354 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:38:51.951546 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:51.951418 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:51.951546 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:51.951487 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:38:53.950316 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:53.950276 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:53.950776 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:53.950276 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:38:53.950776 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:53.950276 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:38:53.950776 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:53.950415 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:38:53.950776 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:53.950516 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:38:53.950776 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:53.950618 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:38:54.452149 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:54.452116 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4ksh\" (UniqueName: \"kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh\") pod \"network-check-target-d6xks\" (UID: \"135de3f6-fc59-4070-9899-88ac1d14ffde\") " pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:38:54.452360 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:54.452328 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:38:54.452360 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:54.452359 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:38:54.452436 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:54.452373 2584 projected.go:196] Error preparing data for projected volume kube-api-access-k4ksh for pod openshift-network-diagnostics/network-check-target-d6xks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:38:54.452471 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:54.452434 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh podName:135de3f6-fc59-4070-9899-88ac1d14ffde nodeName:}" failed. No retries permitted until 2026-07-08 17:39:02.452415671 +0000 UTC m=+18.136512946 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4ksh" (UniqueName: "kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh") pod "network-check-target-d6xks" (UID: "135de3f6-fc59-4070-9899-88ac1d14ffde") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:38:54.553346 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:54.553069 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret\") pod \"global-pull-secret-syncer-gbq5c\" (UID: \"ac3c6d61-29c8-4bc9-82d8-78177eb9045c\") " pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:54.553346 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:54.553130 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs\") pod \"network-metrics-daemon-h6mc5\" (UID: \"a1fba745-34fb-4f41-b257-c6146b4f88fa\") " pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:38:54.553346 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:54.553236 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:38:54.553346 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:54.553262 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:38:54.553346 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:54.553328 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs podName:a1fba745-34fb-4f41-b257-c6146b4f88fa nodeName:}" failed. No retries permitted until 2026-07-08 17:39:02.553307305 +0000 UTC m=+18.237404565 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs") pod "network-metrics-daemon-h6mc5" (UID: "a1fba745-34fb-4f41-b257-c6146b4f88fa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:38:54.553346 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:54.553346 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret podName:ac3c6d61-29c8-4bc9-82d8-78177eb9045c nodeName:}" failed. No retries permitted until 2026-07-08 17:39:02.553336982 +0000 UTC m=+18.237434240 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret") pod "global-pull-secret-syncer-gbq5c" (UID: "ac3c6d61-29c8-4bc9-82d8-78177eb9045c") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:38:55.950022 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:55.949985 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:55.950515 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:55.949998 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:38:55.950515 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:55.950124 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:38:55.950515 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:55.950139 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:38:55.950515 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:55.950281 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:38:55.950515 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:55.950396 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:38:57.950931 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:57.950682 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:57.951477 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:57.950731 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:38:57.951477 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:57.951048 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:38:57.951477 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:57.951118 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:38:57.951477 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:57.950731 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:38:57.951477 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:57.951218 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:38:58.020316 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:58.020062 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" event={"ID":"35cfa5da-654d-4bb7-a263-b39acc0923bb","Type":"ContainerStarted","Data":"3440234c5c0bbf04c242c122c834dfb9217e194b1184e1dbcc6e0357739be35d"} Jul 08 17:38:58.024170 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:58.024138 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal" event={"ID":"66beb382798cae7e66e45d75b5f0b9d7","Type":"ContainerStarted","Data":"6c90670294dda9f99809fcd3dc525723a043e21144480431f223be27be06d90d"} Jul 08 17:38:58.038036 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:58.037983 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4zlvf" podStartSLOduration=2.766856852 podStartE2EDuration="13.037966146s" totalImagesPullingTime="10.271109294s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:38:58.037670612 +0000 UTC m=+13.721767890" watchObservedRunningTime="2026-07-08 17:38:58.037966146 +0000 UTC m=+13.722063425" Jul 08 17:38:59.028924 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:59.028662 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-95k7k" event={"ID":"a751057a-2392-4b54-86da-2fdb58461203","Type":"ContainerStarted","Data":"4a8957e3287cb0f1d8cf0c4b8b1e46e1b2098c04c9e64f61b48b48d1526366cf"} Jul 08 17:38:59.030609 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:59.030542 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" event={"ID":"3a8c9e0e-553f-4e86-af3a-0277d352166c","Type":"ContainerStarted","Data":"6b1399b7d03d3235860299a68a448f07eac7764c346c7ef74eb0e7ec882b7c78"} Jul 08 17:38:59.032057 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:59.032033 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8h4h6" event={"ID":"e8468fb3-5642-43d6-95e5-c524fb532087","Type":"ContainerStarted","Data":"43e91c2a78bc0892fca1695b2ab80473fa1f2910e7384e095806701d6c32ddf0"} Jul 08 17:38:59.033496 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:59.033462 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rhlxp" event={"ID":"747083d6-492d-413c-99e7-364aca1c1cee","Type":"ContainerStarted","Data":"56f19b91a0c58a9c023bd1d8f42d7e5ee48e74269f3b5daf697bc40a392373a4"} Jul 08 17:38:59.034928 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:59.034906 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bl792" event={"ID":"970dfcb3-0c92-4a96-95c9-fe1db4c711d6","Type":"ContainerStarted","Data":"a91f1073328a04ce9d1fa74ba3a01dded0039a6527889f969b28762ef275f49e"} Jul 08 17:38:59.036578 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:59.036556 2584 generic.go:356] "Generic (PLEG): container finished" podID="8119fcfc2e17d9ba7073558077e669dc" containerID="c6c71da450196fc6dd8a9e0a8cc9a1a428f4acf82b07be38666c0942a86ca892" exitCode=0 Jul 08 17:38:59.036682 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:59.036639 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal" event={"ID":"8119fcfc2e17d9ba7073558077e669dc","Type":"ContainerDied","Data":"c6c71da450196fc6dd8a9e0a8cc9a1a428f4acf82b07be38666c0942a86ca892"} Jul 08 17:38:59.038368 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:59.038341 2584 generic.go:356] "Generic (PLEG): container finished" podID="f4e9d179-2384-44e7-afd8-cb22399b829f" containerID="bc402a64a30682b5b407ef109d6ecc367f9203b4e4487dc83780456ba2458a5e" exitCode=0 Jul 08 17:38:59.038452 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:59.038416 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blvkt" event={"ID":"f4e9d179-2384-44e7-afd8-cb22399b829f","Type":"ContainerDied","Data":"bc402a64a30682b5b407ef109d6ecc367f9203b4e4487dc83780456ba2458a5e"} Jul 08 17:38:59.046735 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:59.046696 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-95k7k" podStartSLOduration=3.822988192 podStartE2EDuration="14.046677293s" totalImagesPullingTime="10.223689101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:38:59.046565322 +0000 UTC m=+14.730662600" watchObservedRunningTime="2026-07-08 17:38:59.046677293 +0000 UTC m=+14.730774572" Jul 08 17:38:59.046961 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:59.046931 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-226.ec2.internal" podStartSLOduration=13.046925501 podStartE2EDuration="13.046925501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:38:58.050815709 +0000 UTC m=+13.734912988" watchObservedRunningTime="2026-07-08 17:38:59.046925501 +0000 UTC m=+14.731022779" Jul 08 17:38:59.082399 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:59.082345 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bl792" podStartSLOduration=3.813895248 podStartE2EDuration="14.082326252s" totalImagesPullingTime="10.268431004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:38:59.081665887 +0000 UTC m=+14.765763166" watchObservedRunningTime="2026-07-08 17:38:59.082326252 +0000 UTC m=+14.766423529" Jul 08 17:38:59.126607 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:59.126555 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8h4h6" podStartSLOduration=3.868209237 podStartE2EDuration="14.1265337s" totalImagesPullingTime="10.258324463s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:38:59.101114589 +0000 UTC m=+14.785211866" watchObservedRunningTime="2026-07-08 17:38:59.1265337 +0000 UTC m=+14.810630988" Jul 08 17:38:59.142231 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:59.141766 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rhlxp" podStartSLOduration=3.8730448170000003 podStartE2EDuration="14.141743718s" totalImagesPullingTime="10.268698901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:38:59.141135437 +0000 UTC m=+14.825232715" watchObservedRunningTime="2026-07-08 17:38:59.141743718 +0000 UTC m=+14.825840995" Jul 08 17:38:59.728781 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:59.728724 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bl792" Jul 08 17:38:59.729443 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:59.729425 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bl792" Jul 08 17:38:59.950523 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:59.950059 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:38:59.950523 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:59.950189 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:38:59.950523 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:59.950204 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:38:59.950523 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:59.950335 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:38:59.950523 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:38:59.950398 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:38:59.950523 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:38:59.950487 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:39:00.041936 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:00.041834 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal" event={"ID":"8119fcfc2e17d9ba7073558077e669dc","Type":"ContainerStarted","Data":"55e6033ec68b86344f3d86b74aecff91f94e51d29b76857155872e66d8177c40"} Jul 08 17:39:00.057294 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:00.057227 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-226.ec2.internal" podStartSLOduration=14.057211684 podStartE2EDuration="14.057211684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:39:00.056855659 +0000 UTC m=+15.740952936" watchObservedRunningTime="2026-07-08 17:39:00.057211684 +0000 UTC m=+15.741308960" Jul 08 17:39:00.229699 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:00.229676 2584 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 08 17:39:00.819384 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:00.819263 2584 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T17:39:00.229694533Z","UUID":"8bd6910d-7625-4dcd-88fa-35156c9916e6","Handler":null,"Name":"","Endpoint":""} Jul 08 17:39:00.821168 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:00.821141 2584 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 08 17:39:00.821310 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:00.821182 2584 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 08 17:39:01.046800 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:01.046762 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" event={"ID":"3a8c9e0e-553f-4e86-af3a-0277d352166c","Type":"ContainerStarted","Data":"96b592c2c03dbf648ba41e36ff627d0faa029b0cd4e47f334c0abe57ce2bb6ab"} Jul 08 17:39:01.047488 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:01.046830 2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 08 17:39:01.950773 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:01.950743 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:39:01.951002 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:01.950868 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:39:01.951002 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:01.950960 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:39:01.951106 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:01.951045 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:39:01.951106 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:01.951097 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:01.951169 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:01.951156 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:39:02.517080 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:02.517033 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4ksh\" (UniqueName: \"kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh\") pod \"network-check-target-d6xks\" (UID: \"135de3f6-fc59-4070-9899-88ac1d14ffde\") " pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:02.517604 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:02.517201 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:39:02.517604 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:02.517224 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:39:02.517604 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:02.517251 2584 projected.go:196] Error preparing data for projected volume kube-api-access-k4ksh for pod openshift-network-diagnostics/network-check-target-d6xks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:39:02.517604 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:02.517316 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh podName:135de3f6-fc59-4070-9899-88ac1d14ffde nodeName:}" failed. No retries permitted until 2026-07-08 17:39:18.517296537 +0000 UTC m=+34.201393795 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4ksh" (UniqueName: "kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh") pod "network-check-target-d6xks" (UID: "135de3f6-fc59-4070-9899-88ac1d14ffde") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:39:02.618231 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:02.618188 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret\") pod \"global-pull-secret-syncer-gbq5c\" (UID: \"ac3c6d61-29c8-4bc9-82d8-78177eb9045c\") " pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:39:02.618434 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:02.618262 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs\") pod \"network-metrics-daemon-h6mc5\" (UID: \"a1fba745-34fb-4f41-b257-c6146b4f88fa\") " pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:39:02.618434 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:02.618369 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:39:02.618434 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:02.618433 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs podName:a1fba745-34fb-4f41-b257-c6146b4f88fa nodeName:}" failed. No retries permitted until 2026-07-08 17:39:18.61841508 +0000 UTC m=+34.302512340 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs") pod "network-metrics-daemon-h6mc5" (UID: "a1fba745-34fb-4f41-b257-c6146b4f88fa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:39:02.618604 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:02.618371 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:39:02.618604 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:02.618517 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret podName:ac3c6d61-29c8-4bc9-82d8-78177eb9045c nodeName:}" failed. No retries permitted until 2026-07-08 17:39:18.618498559 +0000 UTC m=+34.302595837 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret") pod "global-pull-secret-syncer-gbq5c" (UID: "ac3c6d61-29c8-4bc9-82d8-78177eb9045c") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:39:03.950592 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:03.950337 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:03.951108 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:03.950348 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:39:03.951108 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:03.950693 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:39:03.951108 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:03.950755 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:39:03.951108 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:03.950357 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:39:03.951108 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:03.950858 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:39:05.950538 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:05.950502 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:39:05.951014 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:05.950505 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:05.951014 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:05.950616 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:39:05.951014 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:05.950506 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:39:05.951014 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:05.950713 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:39:05.951014 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:05.950778 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:39:07.950117 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:07.950088 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:39:07.950693 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:07.950217 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:39:07.950693 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:07.950205 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:39:07.950693 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:07.950277 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:07.950693 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:07.950336 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:39:07.950693 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:07.950396 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:39:09.950151 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:09.950113 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:09.950669 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:09.950235 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:39:09.950669 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:09.950254 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:39:09.950669 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:09.950351 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:39:09.950669 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:09.950256 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:39:09.950669 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:09.950440 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:39:11.602964 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:11.602780 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bl792" Jul 08 17:39:11.603288 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:11.603085 2584 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 08 17:39:11.603793 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:11.603623 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bl792" Jul 08 17:39:11.950334 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:11.950310 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:39:11.950441 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:11.950338 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:39:11.950441 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:11.950310 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:11.950441 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:11.950416 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:39:11.950545 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:11.950498 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:39:11.950593 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:11.950572 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:39:12.072573 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:12.072541 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8zt5s" event={"ID":"67ae88d7-7fa6-494a-9bd0-43c15921cb8c","Type":"ContainerStarted","Data":"ae260d48c44e83cd883e4682562f2917945b1d4005095a85f494e4fe7aa3a206"} Jul 08 17:39:12.074404 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:12.074379 2584 generic.go:356] "Generic (PLEG): container finished" podID="f4e9d179-2384-44e7-afd8-cb22399b829f" containerID="df0d9804fe3e69f6b29dbebbf04bb02666e3a03634f5350a99e021b34ddb7a28" exitCode=0 Jul 08 17:39:12.074512 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:12.074462 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blvkt" event={"ID":"f4e9d179-2384-44e7-afd8-cb22399b829f","Type":"ContainerDied","Data":"df0d9804fe3e69f6b29dbebbf04bb02666e3a03634f5350a99e021b34ddb7a28"} Jul 08 17:39:12.079906 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:12.079884 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-krx82" event={"ID":"3b773719-d2d4-4be0-a119-71ffc944336c","Type":"ContainerStarted","Data":"ec94f4872122ec7260188bab8d1ec80a79cfb5ecf2f028100c118c475aee8012"} Jul 08 17:39:12.079991 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:12.079955 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-krx82" event={"ID":"3b773719-d2d4-4be0-a119-71ffc944336c","Type":"ContainerStarted","Data":"b56c5defdf1c6ab665fdf88281bc6b77c52f0f49bb06366e51e01d00cbb609dd"} Jul 08 17:39:12.080045 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:12.080008 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-krx82" event={"ID":"3b773719-d2d4-4be0-a119-71ffc944336c","Type":"ContainerStarted","Data":"76f63206549b1a6a4cfd9ff13b75a6a06f5c2480683d851248586684ce7bead7"} Jul 08 17:39:12.080093 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:12.080054 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-krx82" event={"ID":"3b773719-d2d4-4be0-a119-71ffc944336c","Type":"ContainerStarted","Data":"0902214c168dcaf60a7474d89d183df98f3f324ed7284b7b5112183af3af0914"} Jul 08 17:39:12.080137 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:12.080096 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-krx82" event={"ID":"3b773719-d2d4-4be0-a119-71ffc944336c","Type":"ContainerStarted","Data":"0092b7453cb41d8d09bb4bad19bfd99a5e1ceaec5937ef072d4389cdbfedaa82"} Jul 08 17:39:12.082288 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:12.082267 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" event={"ID":"3a8c9e0e-553f-4e86-af3a-0277d352166c","Type":"ContainerStarted","Data":"9262e39b2540fc5cf776a654573ae611df77f837b80ffd9a4ac92c53770302f3"} Jul 08 17:39:12.097557 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:12.097508 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8zt5s" podStartSLOduration=2.91832615 podStartE2EDuration="27.097494255s" totalImagesPullingTime="24.179168105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:39:12.097099753 +0000 UTC m=+27.781197030" watchObservedRunningTime="2026-07-08 17:39:12.097494255 +0000 UTC m=+27.781591532" Jul 08 17:39:12.128284 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:12.128215 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-49bpp" podStartSLOduration=3.9736105139999998 podStartE2EDuration="27.128195973s" totalImagesPullingTime="23.154585459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:38:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:39:12.128147832 +0000 UTC m=+27.812245110" watchObservedRunningTime="2026-07-08 17:39:12.128195973 +0000 UTC m=+27.812293278" Jul 08 17:39:13.088094 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:13.088057 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-krx82" event={"ID":"3b773719-d2d4-4be0-a119-71ffc944336c","Type":"ContainerStarted","Data":"fcc03ebd8bb717ca4bdd077c3bd33dce4139b00c614ca49ca288728adcffd747"} Jul 08 17:39:13.950530 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:13.950496 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:39:13.950530 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:13.950530 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:39:13.950756 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:13.950612 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:39:13.950756 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:13.950664 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:13.950824 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:13.950754 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:39:13.950857 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:13.950824 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:39:14.091951 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:14.091914 2584 generic.go:356] "Generic (PLEG): container finished" podID="f4e9d179-2384-44e7-afd8-cb22399b829f" containerID="a9e2ef24bec62662459960bec3e3726549f5e5d900da592cafc9d8e934d9c630" exitCode=0 Jul 08 17:39:14.091951 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:14.091954 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blvkt" event={"ID":"f4e9d179-2384-44e7-afd8-cb22399b829f","Type":"ContainerDied","Data":"a9e2ef24bec62662459960bec3e3726549f5e5d900da592cafc9d8e934d9c630"} Jul 08 17:39:15.097112 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:15.097070 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-krx82" event={"ID":"3b773719-d2d4-4be0-a119-71ffc944336c","Type":"ContainerStarted","Data":"850e9d14c05fcf38ac925ee9245656fc87590884c1ea190bfb539c6dfec27560"} Jul 08 17:39:15.950784 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:15.950756 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:39:15.950948 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:15.950756 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:39:15.950948 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:15.950863 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:39:15.950948 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:15.950756 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:15.951050 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:15.950949 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:39:15.951050 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:15.951011 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:39:16.102165 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:16.102128 2584 generic.go:356] "Generic (PLEG): container finished" podID="f4e9d179-2384-44e7-afd8-cb22399b829f" containerID="196e89971bac8c0e2ee5b15e7d79b68d66be18716085f54c192d0d6d5bd8c192" exitCode=0 Jul 08 17:39:16.102165 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:16.102165 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blvkt" event={"ID":"f4e9d179-2384-44e7-afd8-cb22399b829f","Type":"ContainerDied","Data":"196e89971bac8c0e2ee5b15e7d79b68d66be18716085f54c192d0d6d5bd8c192"} Jul 08 17:39:17.107686 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:17.107499 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-krx82" event={"ID":"3b773719-d2d4-4be0-a119-71ffc944336c","Type":"ContainerStarted","Data":"ce2c806bd50fc0993146ea5cb5b4ba1bad9dcbab1015f14960403348358014e2"} Jul 08 17:39:17.108094 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:17.107736 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:39:17.108094 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:17.107782 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:39:17.108094 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:17.107829 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:39:17.122865 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:17.122841 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:39:17.126674 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:17.126658 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:39:17.137174 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:17.137127 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-krx82" podStartSLOduration=7.803166546 podStartE2EDuration="32.137113469s" totalImagesPullingTime="24.333946923s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:39:17.135963168 +0000 UTC m=+32.820060464" watchObservedRunningTime="2026-07-08 17:39:17.137113469 +0000 UTC m=+32.821210739" Jul 08 17:39:17.950772 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:17.950741 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:39:17.950772 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:17.950766 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:39:17.951008 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:17.950756 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:17.951008 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:17.950858 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:39:17.951008 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:17.950949 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:39:17.951111 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:17.951026 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:39:18.527312 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:18.527281 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4ksh\" (UniqueName: \"kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh\") pod \"network-check-target-d6xks\" (UID: \"135de3f6-fc59-4070-9899-88ac1d14ffde\") " pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:18.527732 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:18.527434 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 17:39:18.527732 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:18.527454 2584 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 17:39:18.527732 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:18.527463 2584 projected.go:196] Error preparing data for projected volume kube-api-access-k4ksh for pod openshift-network-diagnostics/network-check-target-d6xks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:39:18.527732 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:18.527511 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh podName:135de3f6-fc59-4070-9899-88ac1d14ffde nodeName:}" failed. No retries permitted until 2026-07-08 17:39:50.527498055 +0000 UTC m=+66.211595315 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-k4ksh" (UniqueName: "kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh") pod "network-check-target-d6xks" (UID: "135de3f6-fc59-4070-9899-88ac1d14ffde") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 17:39:18.618645 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:18.618617 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gbq5c"] Jul 08 17:39:18.618795 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:18.618741 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:39:18.618836 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:18.618818 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:39:18.621729 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:18.621700 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h6mc5"] Jul 08 17:39:18.621836 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:18.621797 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:39:18.621887 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:18.621872 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:39:18.623979 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:18.623956 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d6xks"] Jul 08 17:39:18.624099 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:18.624034 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:18.624099 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:18.624092 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:39:18.627646 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:18.627620 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret\") pod \"global-pull-secret-syncer-gbq5c\" (UID: \"ac3c6d61-29c8-4bc9-82d8-78177eb9045c\") " pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:39:18.627743 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:18.627658 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs\") pod \"network-metrics-daemon-h6mc5\" (UID: \"a1fba745-34fb-4f41-b257-c6146b4f88fa\") " pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:39:18.627804 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:18.627759 2584 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:39:18.627855 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:18.627806 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs podName:a1fba745-34fb-4f41-b257-c6146b4f88fa nodeName:}" failed. No retries permitted until 2026-07-08 17:39:50.627792966 +0000 UTC m=+66.311890222 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs") pod "network-metrics-daemon-h6mc5" (UID: "a1fba745-34fb-4f41-b257-c6146b4f88fa") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 17:39:18.627855 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:18.627759 2584 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 17:39:18.627855 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:18.627843 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret podName:ac3c6d61-29c8-4bc9-82d8-78177eb9045c nodeName:}" failed. No retries permitted until 2026-07-08 17:39:50.627837192 +0000 UTC m=+66.311934447 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret") pod "global-pull-secret-syncer-gbq5c" (UID: "ac3c6d61-29c8-4bc9-82d8-78177eb9045c") : object "kube-system"/"original-pull-secret" not registered Jul 08 17:39:19.950383 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:19.950353 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:39:19.950794 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:19.950443 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:39:19.950794 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:19.950448 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:39:19.950794 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:19.950469 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:19.950794 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:19.950545 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:39:19.950794 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:19.950617 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:39:21.950586 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:21.950393 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:21.951008 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:21.950403 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:39:21.951008 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:21.950672 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:39:21.951008 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:21.950450 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:39:21.951008 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:21.950745 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:39:21.951008 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:21.950844 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:39:23.950408 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:23.950369 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:39:23.950976 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:23.950369 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:23.950976 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:23.950474 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gbq5c" podUID="ac3c6d61-29c8-4bc9-82d8-78177eb9045c" Jul 08 17:39:23.950976 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:23.950576 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-d6xks" podUID="135de3f6-fc59-4070-9899-88ac1d14ffde" Jul 08 17:39:23.950976 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:23.950369 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:39:23.950976 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:23.950679 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h6mc5" podUID="a1fba745-34fb-4f41-b257-c6146b4f88fa" Jul 08 17:39:24.089772 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.089743 2584 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-226.ec2.internal" event="NodeReady" Jul 08 17:39:24.089956 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.089861 2584 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 08 17:39:24.123895 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.123871 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6"] Jul 08 17:39:24.127293 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.127262 2584 generic.go:356] "Generic (PLEG): container finished" podID="f4e9d179-2384-44e7-afd8-cb22399b829f" containerID="0b5ba58a65ebbd63094345dc18705bdaaed3821513a63e20a5bf9755fbc6aa19" exitCode=0 Jul 08 17:39:24.144011 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.143979 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blvkt" event={"ID":"f4e9d179-2384-44e7-afd8-cb22399b829f","Type":"ContainerDied","Data":"0b5ba58a65ebbd63094345dc18705bdaaed3821513a63e20a5bf9755fbc6aa19"} Jul 08 17:39:24.144152 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.144028 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-76b767f9f7-wwf7r"] Jul 08 17:39:24.144226 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.144161 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" Jul 08 17:39:24.146950 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.146931 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 17:39:24.148171 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.148148 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 17:39:24.148363 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.148347 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 08 17:39:24.148439 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.148353 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 08 17:39:24.148503 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.148353 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-64zpd\"" Jul 08 17:39:24.159955 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.159916 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-97cbdb97b-h88tb"] Jul 08 17:39:24.161545 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.161523 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:24.167353 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.166544 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 17:39:24.167353 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.166572 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 08 17:39:24.167353 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.166603 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 17:39:24.167353 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.166828 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 08 17:39:24.167353 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.166826 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 08 17:39:24.167353 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.167020 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 08 17:39:24.167353 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.167115 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e2388ecf-b94e-4738-9818-4ab78c6d1c80-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-57fr6\" (UID: \"e2388ecf-b94e-4738-9818-4ab78c6d1c80\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" Jul 08 17:39:24.167353 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.167153 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:24.167353 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.167191 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-57fr6\" (UID: \"e2388ecf-b94e-4738-9818-4ab78c6d1c80\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" Jul 08 17:39:24.167353 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.167215 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-default-certificate\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:24.167353 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.167256 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:24.167353 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.167301 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7z5r\" (UniqueName: \"kubernetes.io/projected/e2388ecf-b94e-4738-9818-4ab78c6d1c80-kube-api-access-q7z5r\") pod \"cluster-monitoring-operator-64786d9cbb-57fr6\" (UID: \"e2388ecf-b94e-4738-9818-4ab78c6d1c80\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" Jul 08 17:39:24.167353 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.167323 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-stats-auth\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:24.168012 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.167369 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt6cb\" (UniqueName: \"kubernetes.io/projected/7811ee0a-b7e8-45d3-981f-4f81813aeae7-kube-api-access-dt6cb\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:24.168299 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.168223 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8qxg7\"" Jul 08 17:39:24.190100 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.190073 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-h6pfl"] Jul 08 17:39:24.190568 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.190551 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.192802 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.192781 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 08 17:39:24.193018 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.193002 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 08 17:39:24.193441 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.193423 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-96l64\"" Jul 08 17:39:24.193635 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.193582 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 08 17:39:24.210043 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.210020 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww"] Jul 08 17:39:24.210166 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.210137 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-h6pfl" Jul 08 17:39:24.212459 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.212428 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6ggfw\"" Jul 08 17:39:24.212578 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.212441 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 08 17:39:24.212663 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.212649 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 08 17:39:24.220328 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.220305 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 08 17:39:24.228482 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.228450 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tnrxd"] Jul 08 17:39:24.228606 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.228592 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww" Jul 08 17:39:24.230860 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.230833 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 08 17:39:24.230984 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.230914 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mkl8h\"" Jul 08 17:39:24.231054 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.231041 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 08 17:39:24.231290 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.231273 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 08 17:39:24.246527 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.246496 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-fqtgz"] Jul 08 17:39:24.246687 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.246621 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tnrxd" Jul 08 17:39:24.248577 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.248561 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 08 17:39:24.248907 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.248895 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 08 17:39:24.248950 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.248897 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 08 17:39:24.249130 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.249118 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 08 17:39:24.249172 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.249137 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-58hk5\"" Jul 08 17:39:24.260520 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.260501 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-phwp8"] Jul 08 17:39:24.260619 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.260605 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.263564 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.263537 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 08 17:39:24.263901 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.263881 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 17:39:24.263948 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.263898 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 08 17:39:24.263981 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.263969 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jw6ms\"" Jul 08 17:39:24.264331 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.264313 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 17:39:24.268154 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.268122 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-default-certificate\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:24.268282 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.268260 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:24.268400 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.268378 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7z5r\" (UniqueName: \"kubernetes.io/projected/e2388ecf-b94e-4738-9818-4ab78c6d1c80-kube-api-access-q7z5r\") pod \"cluster-monitoring-operator-64786d9cbb-57fr6\" (UID: \"e2388ecf-b94e-4738-9818-4ab78c6d1c80\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" Jul 08 17:39:24.268495 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.268419 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 17:39:24.268495 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.268421 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-stats-auth\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:24.268495 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.268468 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt6cb\" (UniqueName: \"kubernetes.io/projected/7811ee0a-b7e8-45d3-981f-4f81813aeae7-kube-api-access-dt6cb\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:24.268656 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.268512 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs podName:7811ee0a-b7e8-45d3-981f-4f81813aeae7 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:24.768484459 +0000 UTC m=+40.452581718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs") pod "router-default-76b767f9f7-wwf7r" (UID: "7811ee0a-b7e8-45d3-981f-4f81813aeae7") : secret "router-metrics-certs-default" not found Jul 08 17:39:24.268656 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.268536 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-pg8ww\" (UID: \"9df45c99-7be9-417d-9722-31dfff063d5b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww" Jul 08 17:39:24.268812 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.268658 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e2388ecf-b94e-4738-9818-4ab78c6d1c80-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-57fr6\" (UID: \"e2388ecf-b94e-4738-9818-4ab78c6d1c80\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" Jul 08 17:39:24.268812 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.268757 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psdw2\" (UniqueName: \"kubernetes.io/projected/d9f0ea1a-0f7d-496d-bdc4-b0cf946ba514-kube-api-access-psdw2\") pod \"volume-data-source-validator-5d456fdbcd-h6pfl\" (UID: \"d9f0ea1a-0f7d-496d-bdc4-b0cf946ba514\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-h6pfl" Jul 08 17:39:24.268920 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.268867 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:24.269048 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.268931 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmdgr\" (UniqueName: \"kubernetes.io/projected/9df45c99-7be9-417d-9722-31dfff063d5b-kube-api-access-zmdgr\") pod \"cluster-samples-operator-56fcf86bf-pg8ww\" (UID: \"9df45c99-7be9-417d-9722-31dfff063d5b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww" Jul 08 17:39:24.269103 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.269090 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-57fr6\" (UID: \"e2388ecf-b94e-4738-9818-4ab78c6d1c80\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" Jul 08 17:39:24.269676 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.269511 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle podName:7811ee0a-b7e8-45d3-981f-4f81813aeae7 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:24.769474214 +0000 UTC m=+40.453571490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle") pod "router-default-76b767f9f7-wwf7r" (UID: "7811ee0a-b7e8-45d3-981f-4f81813aeae7") : configmap references non-existent config key: service-ca.crt Jul 08 17:39:24.269676 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.269664 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 17:39:24.269855 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.269722 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls podName:e2388ecf-b94e-4738-9818-4ab78c6d1c80 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:24.76970163 +0000 UTC m=+40.453798885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-57fr6" (UID: "e2388ecf-b94e-4738-9818-4ab78c6d1c80") : secret "cluster-monitoring-operator-tls" not found Jul 08 17:39:24.270029 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.270008 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e2388ecf-b94e-4738-9818-4ab78c6d1c80-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-57fr6\" (UID: \"e2388ecf-b94e-4738-9818-4ab78c6d1c80\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" Jul 08 17:39:24.271558 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.271534 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 08 17:39:24.273662 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.273630 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-default-certificate\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:24.273662 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.273654 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-stats-auth\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:24.276998 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.276967 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-sbh59"] Jul 08 17:39:24.277100 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.277078 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-phwp8" Jul 08 17:39:24.280703 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.280678 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7z5r\" (UniqueName: \"kubernetes.io/projected/e2388ecf-b94e-4738-9818-4ab78c6d1c80-kube-api-access-q7z5r\") pod \"cluster-monitoring-operator-64786d9cbb-57fr6\" (UID: \"e2388ecf-b94e-4738-9818-4ab78c6d1c80\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" Jul 08 17:39:24.281190 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.281023 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt6cb\" (UniqueName: \"kubernetes.io/projected/7811ee0a-b7e8-45d3-981f-4f81813aeae7-kube-api-access-dt6cb\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:24.281190 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.281144 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 08 17:39:24.281380 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.281325 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 08 17:39:24.281617 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.281586 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2k85g\"" Jul 08 17:39:24.281812 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.281795 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 08 17:39:24.282018 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.282001 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 08 17:39:24.296168 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.296142 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6"] Jul 08 17:39:24.296323 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.296182 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-h6pfl"] Jul 08 17:39:24.296323 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.296205 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-fqtgz"] Jul 08 17:39:24.296323 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.296229 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-9hqbp"] Jul 08 17:39:24.296886 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.296863 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-sbh59" Jul 08 17:39:24.299589 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.299565 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5ng7g\"" Jul 08 17:39:24.299686 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.299565 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 08 17:39:24.299686 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.299565 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 08 17:39:24.299812 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.299612 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 08 17:39:24.299874 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.299617 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 08 17:39:24.304524 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.304503 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 08 17:39:24.310219 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.310190 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:24.312475 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.312454 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww"] Jul 08 17:39:24.312579 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.312489 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-phwp8"] Jul 08 17:39:24.312579 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.312505 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-97cbdb97b-h88tb"] Jul 08 17:39:24.312579 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.312526 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hfbxp"] Jul 08 17:39:24.312579 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.312564 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9hqbp" Jul 08 17:39:24.314935 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.314899 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 17:39:24.314935 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.314914 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 17:39:24.315055 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.314984 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-52ljd\"" Jul 08 17:39:24.323840 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.323812 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rn2rl"] Jul 08 17:39:24.323924 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.323843 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hfbxp" Jul 08 17:39:24.325957 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.325943 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 17:39:24.326095 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.326080 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tv27j\"" Jul 08 17:39:24.326142 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.326082 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 17:39:24.326279 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.326267 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 17:39:24.335895 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.335870 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76b767f9f7-wwf7r"] Jul 08 17:39:24.335969 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.335901 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-9hqbp"] Jul 08 17:39:24.335969 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.335923 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tnrxd"] Jul 08 17:39:24.335969 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.335943 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-sbh59"] Jul 08 17:39:24.335969 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.335962 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hfbxp"] Jul 08 17:39:24.336091 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.335986 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rn2rl"] Jul 08 17:39:24.336091 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.335988 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:24.338218 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.338203 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 17:39:24.338356 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.338341 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 17:39:24.338561 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.338547 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2bzl9\"" Jul 08 17:39:24.371174 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371149 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/47839b2a-54b6-4723-bba7-f43ef243c1b1-serving-cert\") pod \"console-operator-68b757fb67-sbh59\" (UID: \"47839b2a-54b6-4723-bba7-f43ef243c1b1\") " pod="openshift-console-operator/console-operator-68b757fb67-sbh59" Jul 08 17:39:24.371315 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371183 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce741a8f-ef63-4a40-a144-a4ce45e2e649-cert\") pod \"ingress-canary-hfbxp\" (UID: \"ce741a8f-ef63-4a40-a144-a4ce45e2e649\") " pod="openshift-ingress-canary/ingress-canary-hfbxp" Jul 08 17:39:24.371315 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371268 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-certificates\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.371315 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371299 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/278faa32-e840-4525-a31d-cf894fdc4b2d-trusted-ca\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.371450 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371321 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnsgr\" (UniqueName: \"kubernetes.io/projected/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-kube-api-access-dnsgr\") pod \"dns-default-rn2rl\" (UID: \"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60\") " pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:24.371450 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371360 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f4hv\" (UniqueName: \"kubernetes.io/projected/05b7e47c-d931-46f1-a21d-fcb853e77030-kube-api-access-8f4hv\") pod \"insights-operator-5c847dcfc4-fqtgz\" (UID: \"05b7e47c-d931-46f1-a21d-fcb853e77030\") " pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.371450 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371376 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wlzb\" (UniqueName: \"kubernetes.io/projected/8b486c24-34df-4756-81fe-95b0e9577762-kube-api-access-2wlzb\") pod \"network-check-source-989f8b8dc-9hqbp\" (UID: \"8b486c24-34df-4756-81fe-95b0e9577762\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9hqbp" Jul 08 17:39:24.371562 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371469 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-pg8ww\" (UID: \"9df45c99-7be9-417d-9722-31dfff063d5b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww" Jul 08 17:39:24.371562 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371494 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/05b7e47c-d931-46f1-a21d-fcb853e77030-snapshots\") pod \"insights-operator-5c847dcfc4-fqtgz\" (UID: \"05b7e47c-d931-46f1-a21d-fcb853e77030\") " pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.371562 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371520 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cca60bdc-e638-44e1-a091-9958e5de0c23-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-tnrxd\" (UID: \"cca60bdc-e638-44e1-a091-9958e5de0c23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tnrxd" Jul 08 17:39:24.371562 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371546 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls\") pod \"dns-default-rn2rl\" (UID: \"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60\") " pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:24.371562 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.371558 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 17:39:24.371751 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371592 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psdw2\" (UniqueName: \"kubernetes.io/projected/d9f0ea1a-0f7d-496d-bdc4-b0cf946ba514-kube-api-access-psdw2\") pod \"volume-data-source-validator-5d456fdbcd-h6pfl\" (UID: \"d9f0ea1a-0f7d-496d-bdc4-b0cf946ba514\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-h6pfl" Jul 08 17:39:24.371751 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.371609 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls podName:9df45c99-7be9-417d-9722-31dfff063d5b nodeName:}" failed. No retries permitted until 2026-07-08 17:39:24.871593808 +0000 UTC m=+40.555691064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-pg8ww" (UID: "9df45c99-7be9-417d-9722-31dfff063d5b") : secret "samples-operator-tls" not found Jul 08 17:39:24.371751 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371645 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmdgr\" (UniqueName: \"kubernetes.io/projected/9df45c99-7be9-417d-9722-31dfff063d5b-kube-api-access-zmdgr\") pod \"cluster-samples-operator-56fcf86bf-pg8ww\" (UID: \"9df45c99-7be9-417d-9722-31dfff063d5b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww" Jul 08 17:39:24.371751 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371666 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05b7e47c-d931-46f1-a21d-fcb853e77030-serving-cert\") pod \"insights-operator-5c847dcfc4-fqtgz\" (UID: \"05b7e47c-d931-46f1-a21d-fcb853e77030\") " pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.371751 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371682 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28cm7\" (UniqueName: \"kubernetes.io/projected/ce741a8f-ef63-4a40-a144-a4ce45e2e649-kube-api-access-28cm7\") pod \"ingress-canary-hfbxp\" (UID: \"ce741a8f-ef63-4a40-a144-a4ce45e2e649\") " pod="openshift-ingress-canary/ingress-canary-hfbxp" Jul 08 17:39:24.371751 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371698 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05b7e47c-d931-46f1-a21d-fcb853e77030-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-fqtgz\" (UID: \"05b7e47c-d931-46f1-a21d-fcb853e77030\") " pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.371751 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371720 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d9fbde7-7603-40a6-9fd9-489246aa05bc-config\") pod \"service-ca-operator-5d46d9ffd4-phwp8\" (UID: \"9d9fbde7-7603-40a6-9fd9-489246aa05bc\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-phwp8" Jul 08 17:39:24.372075 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371759 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-tmp-dir\") pod \"dns-default-rn2rl\" (UID: \"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60\") " pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:24.372075 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371811 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/278faa32-e840-4525-a31d-cf894fdc4b2d-installation-pull-secrets\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.372075 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371842 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cca60bdc-e638-44e1-a091-9958e5de0c23-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-tnrxd\" (UID: \"cca60bdc-e638-44e1-a091-9958e5de0c23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tnrxd" Jul 08 17:39:24.372075 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371876 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-bound-sa-token\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.372075 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371903 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47839b2a-54b6-4723-bba7-f43ef243c1b1-config\") pod \"console-operator-68b757fb67-sbh59\" (UID: \"47839b2a-54b6-4723-bba7-f43ef243c1b1\") " pod="openshift-console-operator/console-operator-68b757fb67-sbh59" Jul 08 17:39:24.372075 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371924 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05b7e47c-d931-46f1-a21d-fcb853e77030-tmp\") pod \"insights-operator-5c847dcfc4-fqtgz\" (UID: \"05b7e47c-d931-46f1-a21d-fcb853e77030\") " pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.372075 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371946 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.372075 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371961 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9d9fbde7-7603-40a6-9fd9-489246aa05bc-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-phwp8\" (UID: \"9d9fbde7-7603-40a6-9fd9-489246aa05bc\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-phwp8" Jul 08 17:39:24.372075 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371975 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jhw8\" (UniqueName: \"kubernetes.io/projected/9d9fbde7-7603-40a6-9fd9-489246aa05bc-kube-api-access-8jhw8\") pod \"service-ca-operator-5d46d9ffd4-phwp8\" (UID: \"9d9fbde7-7603-40a6-9fd9-489246aa05bc\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-phwp8" Jul 08 17:39:24.372075 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.371994 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjp7r\" (UniqueName: \"kubernetes.io/projected/cca60bdc-e638-44e1-a091-9958e5de0c23-kube-api-access-xjp7r\") pod \"kube-storage-version-migrator-operator-846bf668f7-tnrxd\" (UID: \"cca60bdc-e638-44e1-a091-9958e5de0c23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tnrxd" Jul 08 17:39:24.372075 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.372023 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62jpw\" (UniqueName: \"kubernetes.io/projected/47839b2a-54b6-4723-bba7-f43ef243c1b1-kube-api-access-62jpw\") pod \"console-operator-68b757fb67-sbh59\" (UID: \"47839b2a-54b6-4723-bba7-f43ef243c1b1\") " pod="openshift-console-operator/console-operator-68b757fb67-sbh59" Jul 08 17:39:24.372075 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.372047 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-config-volume\") pod \"dns-default-rn2rl\" (UID: \"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60\") " pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:24.372420 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.372096 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/278faa32-e840-4525-a31d-cf894fdc4b2d-image-registry-private-configuration\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.372420 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.372114 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/278faa32-e840-4525-a31d-cf894fdc4b2d-ca-trust-extracted\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.372420 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.372129 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhmr7\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-kube-api-access-mhmr7\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.372420 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.372145 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05b7e47c-d931-46f1-a21d-fcb853e77030-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-fqtgz\" (UID: \"05b7e47c-d931-46f1-a21d-fcb853e77030\") " pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.372420 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.372161 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/47839b2a-54b6-4723-bba7-f43ef243c1b1-trusted-ca\") pod \"console-operator-68b757fb67-sbh59\" (UID: \"47839b2a-54b6-4723-bba7-f43ef243c1b1\") " pod="openshift-console-operator/console-operator-68b757fb67-sbh59" Jul 08 17:39:24.383686 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.383664 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psdw2\" (UniqueName: \"kubernetes.io/projected/d9f0ea1a-0f7d-496d-bdc4-b0cf946ba514-kube-api-access-psdw2\") pod \"volume-data-source-validator-5d456fdbcd-h6pfl\" (UID: \"d9f0ea1a-0f7d-496d-bdc4-b0cf946ba514\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-h6pfl" Jul 08 17:39:24.383766 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.383753 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmdgr\" (UniqueName: \"kubernetes.io/projected/9df45c99-7be9-417d-9722-31dfff063d5b-kube-api-access-zmdgr\") pod \"cluster-samples-operator-56fcf86bf-pg8ww\" (UID: \"9df45c99-7be9-417d-9722-31dfff063d5b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww" Jul 08 17:39:24.397312 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.397264 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:24.472515 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.472477 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/278faa32-e840-4525-a31d-cf894fdc4b2d-image-registry-private-configuration\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.472515 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.472514 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/278faa32-e840-4525-a31d-cf894fdc4b2d-ca-trust-extracted\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.472736 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.472538 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhmr7\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-kube-api-access-mhmr7\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.472736 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.472564 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05b7e47c-d931-46f1-a21d-fcb853e77030-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-fqtgz\" (UID: \"05b7e47c-d931-46f1-a21d-fcb853e77030\") " pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.472736 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.472678 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/47839b2a-54b6-4723-bba7-f43ef243c1b1-trusted-ca\") pod \"console-operator-68b757fb67-sbh59\" (UID: \"47839b2a-54b6-4723-bba7-f43ef243c1b1\") " pod="openshift-console-operator/console-operator-68b757fb67-sbh59" Jul 08 17:39:24.472870 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.472747 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/47839b2a-54b6-4723-bba7-f43ef243c1b1-serving-cert\") pod \"console-operator-68b757fb67-sbh59\" (UID: \"47839b2a-54b6-4723-bba7-f43ef243c1b1\") " pod="openshift-console-operator/console-operator-68b757fb67-sbh59" Jul 08 17:39:24.472870 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.472774 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce741a8f-ef63-4a40-a144-a4ce45e2e649-cert\") pod \"ingress-canary-hfbxp\" (UID: \"ce741a8f-ef63-4a40-a144-a4ce45e2e649\") " pod="openshift-ingress-canary/ingress-canary-hfbxp" Jul 08 17:39:24.472870 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.472824 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-certificates\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.473199 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.473168 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05b7e47c-d931-46f1-a21d-fcb853e77030-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-fqtgz\" (UID: \"05b7e47c-d931-46f1-a21d-fcb853e77030\") " pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.473199 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.473187 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/278faa32-e840-4525-a31d-cf894fdc4b2d-trusted-ca\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.475015 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.472957 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/278faa32-e840-4525-a31d-cf894fdc4b2d-ca-trust-extracted\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.475015 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.473234 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnsgr\" (UniqueName: \"kubernetes.io/projected/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-kube-api-access-dnsgr\") pod \"dns-default-rn2rl\" (UID: \"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60\") " pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:24.475015 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.473338 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f4hv\" (UniqueName: \"kubernetes.io/projected/05b7e47c-d931-46f1-a21d-fcb853e77030-kube-api-access-8f4hv\") pod \"insights-operator-5c847dcfc4-fqtgz\" (UID: \"05b7e47c-d931-46f1-a21d-fcb853e77030\") " pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.475015 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.473375 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wlzb\" (UniqueName: \"kubernetes.io/projected/8b486c24-34df-4756-81fe-95b0e9577762-kube-api-access-2wlzb\") pod \"network-check-source-989f8b8dc-9hqbp\" (UID: \"8b486c24-34df-4756-81fe-95b0e9577762\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9hqbp" Jul 08 17:39:24.475015 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.473468 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-certificates\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.475015 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.473600 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/05b7e47c-d931-46f1-a21d-fcb853e77030-snapshots\") pod \"insights-operator-5c847dcfc4-fqtgz\" (UID: \"05b7e47c-d931-46f1-a21d-fcb853e77030\") " pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.475015 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.473652 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cca60bdc-e638-44e1-a091-9958e5de0c23-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-tnrxd\" (UID: \"cca60bdc-e638-44e1-a091-9958e5de0c23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tnrxd" Jul 08 17:39:24.475015 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.473702 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls\") pod \"dns-default-rn2rl\" (UID: \"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60\") " pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:24.475015 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.473754 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05b7e47c-d931-46f1-a21d-fcb853e77030-serving-cert\") pod \"insights-operator-5c847dcfc4-fqtgz\" (UID: \"05b7e47c-d931-46f1-a21d-fcb853e77030\") " pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.475015 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.473784 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28cm7\" (UniqueName: \"kubernetes.io/projected/ce741a8f-ef63-4a40-a144-a4ce45e2e649-kube-api-access-28cm7\") pod \"ingress-canary-hfbxp\" (UID: \"ce741a8f-ef63-4a40-a144-a4ce45e2e649\") " pod="openshift-ingress-canary/ingress-canary-hfbxp" Jul 08 17:39:24.475015 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.473788 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/47839b2a-54b6-4723-bba7-f43ef243c1b1-trusted-ca\") pod \"console-operator-68b757fb67-sbh59\" (UID: \"47839b2a-54b6-4723-bba7-f43ef243c1b1\") " pod="openshift-console-operator/console-operator-68b757fb67-sbh59" Jul 08 17:39:24.475015 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.473812 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05b7e47c-d931-46f1-a21d-fcb853e77030-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-fqtgz\" (UID: \"05b7e47c-d931-46f1-a21d-fcb853e77030\") " pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.475015 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.473838 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d9fbde7-7603-40a6-9fd9-489246aa05bc-config\") pod \"service-ca-operator-5d46d9ffd4-phwp8\" (UID: \"9d9fbde7-7603-40a6-9fd9-489246aa05bc\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-phwp8" Jul 08 17:39:24.475015 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.473879 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-tmp-dir\") pod \"dns-default-rn2rl\" (UID: \"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60\") " pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:24.475015 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.473909 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/278faa32-e840-4525-a31d-cf894fdc4b2d-installation-pull-secrets\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.475015 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.473934 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cca60bdc-e638-44e1-a091-9958e5de0c23-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-tnrxd\" (UID: \"cca60bdc-e638-44e1-a091-9958e5de0c23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tnrxd" Jul 08 17:39:24.475918 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.473963 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-bound-sa-token\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.475918 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.473996 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47839b2a-54b6-4723-bba7-f43ef243c1b1-config\") pod \"console-operator-68b757fb67-sbh59\" (UID: \"47839b2a-54b6-4723-bba7-f43ef243c1b1\") " pod="openshift-console-operator/console-operator-68b757fb67-sbh59" Jul 08 17:39:24.475918 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.474023 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05b7e47c-d931-46f1-a21d-fcb853e77030-tmp\") pod \"insights-operator-5c847dcfc4-fqtgz\" (UID: \"05b7e47c-d931-46f1-a21d-fcb853e77030\") " pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.475918 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.474049 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.475918 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.474072 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9d9fbde7-7603-40a6-9fd9-489246aa05bc-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-phwp8\" (UID: \"9d9fbde7-7603-40a6-9fd9-489246aa05bc\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-phwp8" Jul 08 17:39:24.475918 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.474096 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jhw8\" (UniqueName: \"kubernetes.io/projected/9d9fbde7-7603-40a6-9fd9-489246aa05bc-kube-api-access-8jhw8\") pod \"service-ca-operator-5d46d9ffd4-phwp8\" (UID: \"9d9fbde7-7603-40a6-9fd9-489246aa05bc\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-phwp8" Jul 08 17:39:24.475918 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.474127 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjp7r\" (UniqueName: \"kubernetes.io/projected/cca60bdc-e638-44e1-a091-9958e5de0c23-kube-api-access-xjp7r\") pod \"kube-storage-version-migrator-operator-846bf668f7-tnrxd\" (UID: \"cca60bdc-e638-44e1-a091-9958e5de0c23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tnrxd" Jul 08 17:39:24.475918 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.474137 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/05b7e47c-d931-46f1-a21d-fcb853e77030-snapshots\") pod \"insights-operator-5c847dcfc4-fqtgz\" (UID: \"05b7e47c-d931-46f1-a21d-fcb853e77030\") " pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.475918 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.474156 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62jpw\" (UniqueName: \"kubernetes.io/projected/47839b2a-54b6-4723-bba7-f43ef243c1b1-kube-api-access-62jpw\") pod \"console-operator-68b757fb67-sbh59\" (UID: \"47839b2a-54b6-4723-bba7-f43ef243c1b1\") " pod="openshift-console-operator/console-operator-68b757fb67-sbh59" Jul 08 17:39:24.475918 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.474182 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-config-volume\") pod \"dns-default-rn2rl\" (UID: \"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60\") " pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:24.475918 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.474190 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/278faa32-e840-4525-a31d-cf894fdc4b2d-trusted-ca\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.475918 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.474498 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cca60bdc-e638-44e1-a091-9958e5de0c23-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-tnrxd\" (UID: \"cca60bdc-e638-44e1-a091-9958e5de0c23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tnrxd" Jul 08 17:39:24.475918 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.474997 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47839b2a-54b6-4723-bba7-f43ef243c1b1-config\") pod \"console-operator-68b757fb67-sbh59\" (UID: \"47839b2a-54b6-4723-bba7-f43ef243c1b1\") " pod="openshift-console-operator/console-operator-68b757fb67-sbh59" Jul 08 17:39:24.475918 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.475040 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05b7e47c-d931-46f1-a21d-fcb853e77030-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-fqtgz\" (UID: \"05b7e47c-d931-46f1-a21d-fcb853e77030\") " pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.475918 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.475182 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d9fbde7-7603-40a6-9fd9-489246aa05bc-config\") pod \"service-ca-operator-5d46d9ffd4-phwp8\" (UID: \"9d9fbde7-7603-40a6-9fd9-489246aa05bc\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-phwp8" Jul 08 17:39:24.475918 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.475455 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 17:39:24.476705 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.475523 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls podName:e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:24.975504035 +0000 UTC m=+40.659601302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls") pod "dns-default-rn2rl" (UID: "e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60") : secret "dns-default-metrics-tls" not found Jul 08 17:39:24.476705 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.475552 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-config-volume\") pod \"dns-default-rn2rl\" (UID: \"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60\") " pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:24.476705 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.476661 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ce741a8f-ef63-4a40-a144-a4ce45e2e649-cert\") pod \"ingress-canary-hfbxp\" (UID: \"ce741a8f-ef63-4a40-a144-a4ce45e2e649\") " pod="openshift-ingress-canary/ingress-canary-hfbxp" Jul 08 17:39:24.476705 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.476683 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05b7e47c-d931-46f1-a21d-fcb853e77030-tmp\") pod \"insights-operator-5c847dcfc4-fqtgz\" (UID: \"05b7e47c-d931-46f1-a21d-fcb853e77030\") " pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.476943 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.476802 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/278faa32-e840-4525-a31d-cf894fdc4b2d-image-registry-private-configuration\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.476943 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.476843 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-tmp-dir\") pod \"dns-default-rn2rl\" (UID: \"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60\") " pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:24.477193 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.477163 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/05b7e47c-d931-46f1-a21d-fcb853e77030-serving-cert\") pod \"insights-operator-5c847dcfc4-fqtgz\" (UID: \"05b7e47c-d931-46f1-a21d-fcb853e77030\") " pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.477317 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.477198 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/47839b2a-54b6-4723-bba7-f43ef243c1b1-serving-cert\") pod \"console-operator-68b757fb67-sbh59\" (UID: \"47839b2a-54b6-4723-bba7-f43ef243c1b1\") " pod="openshift-console-operator/console-operator-68b757fb67-sbh59" Jul 08 17:39:24.477378 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.477350 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 17:39:24.477425 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.477372 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9d9fbde7-7603-40a6-9fd9-489246aa05bc-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-phwp8\" (UID: \"9d9fbde7-7603-40a6-9fd9-489246aa05bc\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-phwp8" Jul 08 17:39:24.477479 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.477369 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97cbdb97b-h88tb: secret "image-registry-tls" not found Jul 08 17:39:24.477526 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.477505 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls podName:278faa32-e840-4525-a31d-cf894fdc4b2d nodeName:}" failed. No retries permitted until 2026-07-08 17:39:24.97748011 +0000 UTC m=+40.661577382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls") pod "image-registry-97cbdb97b-h88tb" (UID: "278faa32-e840-4525-a31d-cf894fdc4b2d") : secret "image-registry-tls" not found Jul 08 17:39:24.481562 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.481542 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/278faa32-e840-4525-a31d-cf894fdc4b2d-installation-pull-secrets\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.482836 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.482814 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cca60bdc-e638-44e1-a091-9958e5de0c23-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-tnrxd\" (UID: \"cca60bdc-e638-44e1-a091-9958e5de0c23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tnrxd" Jul 08 17:39:24.487900 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.487871 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnsgr\" (UniqueName: \"kubernetes.io/projected/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-kube-api-access-dnsgr\") pod \"dns-default-rn2rl\" (UID: \"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60\") " pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:24.488021 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.487941 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28cm7\" (UniqueName: \"kubernetes.io/projected/ce741a8f-ef63-4a40-a144-a4ce45e2e649-kube-api-access-28cm7\") pod \"ingress-canary-hfbxp\" (UID: \"ce741a8f-ef63-4a40-a144-a4ce45e2e649\") " pod="openshift-ingress-canary/ingress-canary-hfbxp" Jul 08 17:39:24.488021 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.487975 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhmr7\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-kube-api-access-mhmr7\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.488695 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.488598 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-bound-sa-token\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.488816 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.488737 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f4hv\" (UniqueName: \"kubernetes.io/projected/05b7e47c-d931-46f1-a21d-fcb853e77030-kube-api-access-8f4hv\") pod \"insights-operator-5c847dcfc4-fqtgz\" (UID: \"05b7e47c-d931-46f1-a21d-fcb853e77030\") " pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.489813 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.489793 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjp7r\" (UniqueName: \"kubernetes.io/projected/cca60bdc-e638-44e1-a091-9958e5de0c23-kube-api-access-xjp7r\") pod \"kube-storage-version-migrator-operator-846bf668f7-tnrxd\" (UID: \"cca60bdc-e638-44e1-a091-9958e5de0c23\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tnrxd" Jul 08 17:39:24.489897 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.489851 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wlzb\" (UniqueName: \"kubernetes.io/projected/8b486c24-34df-4756-81fe-95b0e9577762-kube-api-access-2wlzb\") pod \"network-check-source-989f8b8dc-9hqbp\" (UID: \"8b486c24-34df-4756-81fe-95b0e9577762\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9hqbp" Jul 08 17:39:24.490012 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.489996 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62jpw\" (UniqueName: \"kubernetes.io/projected/47839b2a-54b6-4723-bba7-f43ef243c1b1-kube-api-access-62jpw\") pod \"console-operator-68b757fb67-sbh59\" (UID: \"47839b2a-54b6-4723-bba7-f43ef243c1b1\") " pod="openshift-console-operator/console-operator-68b757fb67-sbh59" Jul 08 17:39:24.490095 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.490077 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jhw8\" (UniqueName: \"kubernetes.io/projected/9d9fbde7-7603-40a6-9fd9-489246aa05bc-kube-api-access-8jhw8\") pod \"service-ca-operator-5d46d9ffd4-phwp8\" (UID: \"9d9fbde7-7603-40a6-9fd9-489246aa05bc\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-phwp8" Jul 08 17:39:24.533111 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.533078 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-h6pfl" Jul 08 17:39:24.562431 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.562402 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tnrxd" Jul 08 17:39:24.573531 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.573509 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" Jul 08 17:39:24.594554 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.594528 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-phwp8" Jul 08 17:39:24.607052 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.607024 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-sbh59" Jul 08 17:39:24.681189 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.680740 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9hqbp" Jul 08 17:39:24.694323 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.694231 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hfbxp" Jul 08 17:39:24.779885 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.778511 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:24.779885 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.778572 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:24.779885 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.778658 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-57fr6\" (UID: \"e2388ecf-b94e-4738-9818-4ab78c6d1c80\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" Jul 08 17:39:24.779885 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.778855 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 17:39:24.779885 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.778928 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls podName:e2388ecf-b94e-4738-9818-4ab78c6d1c80 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:25.778900514 +0000 UTC m=+41.462997782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-57fr6" (UID: "e2388ecf-b94e-4738-9818-4ab78c6d1c80") : secret "cluster-monitoring-operator-tls" not found Jul 08 17:39:24.779885 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.779394 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle podName:7811ee0a-b7e8-45d3-981f-4f81813aeae7 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:25.779377944 +0000 UTC m=+41.463475201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle") pod "router-default-76b767f9f7-wwf7r" (UID: "7811ee0a-b7e8-45d3-981f-4f81813aeae7") : configmap references non-existent config key: service-ca.crt Jul 08 17:39:24.779885 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.779465 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 17:39:24.779885 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.779498 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs podName:7811ee0a-b7e8-45d3-981f-4f81813aeae7 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:25.779487708 +0000 UTC m=+41.463584965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs") pod "router-default-76b767f9f7-wwf7r" (UID: "7811ee0a-b7e8-45d3-981f-4f81813aeae7") : secret "router-metrics-certs-default" not found Jul 08 17:39:24.842959 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.842909 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-h6pfl"] Jul 08 17:39:24.848761 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:39:24.847177 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f0ea1a_0f7d_496d_bdc4_b0cf946ba514.slice/crio-2c59141c88a292610dc4d7e18df35ab856450de876500b036c8a3fee85e1bcdb WatchSource:0}: Error finding container 2c59141c88a292610dc4d7e18df35ab856450de876500b036c8a3fee85e1bcdb: Status 404 returned error can't find the container with id 2c59141c88a292610dc4d7e18df35ab856450de876500b036c8a3fee85e1bcdb Jul 08 17:39:24.849369 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.849300 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-fqtgz"] Jul 08 17:39:24.853068 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:39:24.853036 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b7e47c_d931_46f1_a21d_fcb853e77030.slice/crio-3472d153cdad55298689beac63eb6903c3482c8f8e1af4d0ff4d917760160c8a WatchSource:0}: Error finding container 3472d153cdad55298689beac63eb6903c3482c8f8e1af4d0ff4d917760160c8a: Status 404 returned error can't find the container with id 3472d153cdad55298689beac63eb6903c3482c8f8e1af4d0ff4d917760160c8a Jul 08 17:39:24.860507 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.859792 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tnrxd"] Jul 08 17:39:24.866099 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:39:24.866063 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca60bdc_e638_44e1_a091_9958e5de0c23.slice/crio-11136db53f85ae9d40aa1c5d6dbbf2ecc5bef611ad7010e94b037acddc420e07 WatchSource:0}: Error finding container 11136db53f85ae9d40aa1c5d6dbbf2ecc5bef611ad7010e94b037acddc420e07: Status 404 returned error can't find the container with id 11136db53f85ae9d40aa1c5d6dbbf2ecc5bef611ad7010e94b037acddc420e07 Jul 08 17:39:24.877610 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.877584 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-phwp8"] Jul 08 17:39:24.879217 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.879195 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-pg8ww\" (UID: \"9df45c99-7be9-417d-9722-31dfff063d5b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww" Jul 08 17:39:24.879440 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.879403 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 17:39:24.879545 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.879491 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls podName:9df45c99-7be9-417d-9722-31dfff063d5b nodeName:}" failed. No retries permitted until 2026-07-08 17:39:25.879468224 +0000 UTC m=+41.563565489 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-pg8ww" (UID: "9df45c99-7be9-417d-9722-31dfff063d5b") : secret "samples-operator-tls" not found Jul 08 17:39:24.880423 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:39:24.880389 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9fbde7_7603_40a6_9fd9_489246aa05bc.slice/crio-e509b3f7a4c99332a5e2197a594ec1a1f8127376248bfdfcaf37572a350645a8 WatchSource:0}: Error finding container e509b3f7a4c99332a5e2197a594ec1a1f8127376248bfdfcaf37572a350645a8: Status 404 returned error can't find the container with id e509b3f7a4c99332a5e2197a594ec1a1f8127376248bfdfcaf37572a350645a8 Jul 08 17:39:24.906381 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.906306 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-sbh59"] Jul 08 17:39:24.906519 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.906471 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:24.909576 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:39:24.909554 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47839b2a_54b6_4723_bba7_f43ef243c1b1.slice/crio-48146f9fcbd54b57f88a5d744815dc99a335aad3700db70473469daafa69f2db WatchSource:0}: Error finding container 48146f9fcbd54b57f88a5d744815dc99a335aad3700db70473469daafa69f2db: Status 404 returned error can't find the container with id 48146f9fcbd54b57f88a5d744815dc99a335aad3700db70473469daafa69f2db Jul 08 17:39:24.921628 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.921608 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-9hqbp"] Jul 08 17:39:24.924347 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:39:24.924319 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b486c24_34df_4756_81fe_95b0e9577762.slice/crio-f9f18ea5bd6eaa9f94d2ae701100d913795fc0034ecb9181ea43225212034f26 WatchSource:0}: Error finding container f9f18ea5bd6eaa9f94d2ae701100d913795fc0034ecb9181ea43225212034f26: Status 404 returned error can't find the container with id f9f18ea5bd6eaa9f94d2ae701100d913795fc0034ecb9181ea43225212034f26 Jul 08 17:39:24.934592 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.934568 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hfbxp"] Jul 08 17:39:24.938132 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:39:24.938108 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce741a8f_ef63_4a40_a144_a4ce45e2e649.slice/crio-752f3bab476eaf1802a03639bb70e686e6e7e019e36dcc0caa17944f1bd66e3a WatchSource:0}: Error finding container 752f3bab476eaf1802a03639bb70e686e6e7e019e36dcc0caa17944f1bd66e3a: Status 404 returned error can't find the container with id 752f3bab476eaf1802a03639bb70e686e6e7e019e36dcc0caa17944f1bd66e3a Jul 08 17:39:24.980439 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.980400 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls\") pod \"dns-default-rn2rl\" (UID: \"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60\") " pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:24.981050 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.980476 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:24.981050 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.980545 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 17:39:24.981050 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.980607 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls podName:e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:25.980592887 +0000 UTC m=+41.664690142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls") pod "dns-default-rn2rl" (UID: "e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60") : secret "dns-default-metrics-tls" not found Jul 08 17:39:24.981050 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.980623 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 17:39:24.981050 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.980636 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97cbdb97b-h88tb: secret "image-registry-tls" not found Jul 08 17:39:24.981050 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:24.980679 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls podName:278faa32-e840-4525-a31d-cf894fdc4b2d nodeName:}" failed. No retries permitted until 2026-07-08 17:39:25.980664732 +0000 UTC m=+41.664762006 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls") pod "image-registry-97cbdb97b-h88tb" (UID: "278faa32-e840-4525-a31d-cf894fdc4b2d") : secret "image-registry-tls" not found Jul 08 17:39:24.991057 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:24.991022 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:25.131127 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.131091 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-sbh59" event={"ID":"47839b2a-54b6-4723-bba7-f43ef243c1b1","Type":"ContainerStarted","Data":"48146f9fcbd54b57f88a5d744815dc99a335aad3700db70473469daafa69f2db"} Jul 08 17:39:25.133873 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.133848 2584 generic.go:356] "Generic (PLEG): container finished" podID="f4e9d179-2384-44e7-afd8-cb22399b829f" containerID="3e9ac92d291d4a18afb73c904d44be4324964612b9d663d44b94ff663730e994" exitCode=0 Jul 08 17:39:25.133985 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.133921 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blvkt" event={"ID":"f4e9d179-2384-44e7-afd8-cb22399b829f","Type":"ContainerDied","Data":"3e9ac92d291d4a18afb73c904d44be4324964612b9d663d44b94ff663730e994"} Jul 08 17:39:25.135044 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.135007 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hfbxp" event={"ID":"ce741a8f-ef63-4a40-a144-a4ce45e2e649","Type":"ContainerStarted","Data":"752f3bab476eaf1802a03639bb70e686e6e7e019e36dcc0caa17944f1bd66e3a"} Jul 08 17:39:25.136055 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.136035 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-phwp8" event={"ID":"9d9fbde7-7603-40a6-9fd9-489246aa05bc","Type":"ContainerStarted","Data":"e509b3f7a4c99332a5e2197a594ec1a1f8127376248bfdfcaf37572a350645a8"} Jul 08 17:39:25.138662 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.138625 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tnrxd" event={"ID":"cca60bdc-e638-44e1-a091-9958e5de0c23","Type":"ContainerStarted","Data":"11136db53f85ae9d40aa1c5d6dbbf2ecc5bef611ad7010e94b037acddc420e07"} Jul 08 17:39:25.140469 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.140449 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-h6pfl" event={"ID":"d9f0ea1a-0f7d-496d-bdc4-b0cf946ba514","Type":"ContainerStarted","Data":"2c59141c88a292610dc4d7e18df35ab856450de876500b036c8a3fee85e1bcdb"} Jul 08 17:39:25.145078 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.145050 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9hqbp" event={"ID":"8b486c24-34df-4756-81fe-95b0e9577762","Type":"ContainerStarted","Data":"f9f18ea5bd6eaa9f94d2ae701100d913795fc0034ecb9181ea43225212034f26"} Jul 08 17:39:25.145987 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.145967 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" event={"ID":"05b7e47c-d931-46f1-a21d-fcb853e77030","Type":"ContainerStarted","Data":"3472d153cdad55298689beac63eb6903c3482c8f8e1af4d0ff4d917760160c8a"} Jul 08 17:39:25.789551 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.789507 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-57fr6\" (UID: \"e2388ecf-b94e-4738-9818-4ab78c6d1c80\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" Jul 08 17:39:25.789754 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.789643 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:25.789754 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.789687 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:25.789880 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:25.789864 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 17:39:25.790287 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:25.789933 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs podName:7811ee0a-b7e8-45d3-981f-4f81813aeae7 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:27.789913396 +0000 UTC m=+43.474010654 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs") pod "router-default-76b767f9f7-wwf7r" (UID: "7811ee0a-b7e8-45d3-981f-4f81813aeae7") : secret "router-metrics-certs-default" not found Jul 08 17:39:25.790287 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:25.790096 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle podName:7811ee0a-b7e8-45d3-981f-4f81813aeae7 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:27.790074045 +0000 UTC m=+43.474171314 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle") pod "router-default-76b767f9f7-wwf7r" (UID: "7811ee0a-b7e8-45d3-981f-4f81813aeae7") : configmap references non-existent config key: service-ca.crt Jul 08 17:39:25.790287 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:25.790178 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 17:39:25.790287 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:25.790216 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls podName:e2388ecf-b94e-4738-9818-4ab78c6d1c80 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:27.79020377 +0000 UTC m=+43.474301029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-57fr6" (UID: "e2388ecf-b94e-4738-9818-4ab78c6d1c80") : secret "cluster-monitoring-operator-tls" not found Jul 08 17:39:25.890951 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.890784 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-pg8ww\" (UID: \"9df45c99-7be9-417d-9722-31dfff063d5b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww" Jul 08 17:39:25.891119 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:25.890971 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 17:39:25.891119 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:25.891044 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls podName:9df45c99-7be9-417d-9722-31dfff063d5b nodeName:}" failed. No retries permitted until 2026-07-08 17:39:27.891025601 +0000 UTC m=+43.575122859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-pg8ww" (UID: "9df45c99-7be9-417d-9722-31dfff063d5b") : secret "samples-operator-tls" not found Jul 08 17:39:25.951584 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.950169 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:25.951584 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.950647 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:39:25.951584 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.951056 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:39:25.955835 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.955445 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 17:39:25.955835 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.955674 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 17:39:25.956980 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.956235 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d5skf\"" Jul 08 17:39:25.956980 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.956469 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jwhsf\"" Jul 08 17:39:25.993611 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.992610 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls\") pod \"dns-default-rn2rl\" (UID: \"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60\") " pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:25.993611 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:25.992694 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:25.993611 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:25.992860 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 17:39:25.993611 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:25.992875 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97cbdb97b-h88tb: secret "image-registry-tls" not found Jul 08 17:39:25.993611 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:25.992931 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls podName:278faa32-e840-4525-a31d-cf894fdc4b2d nodeName:}" failed. No retries permitted until 2026-07-08 17:39:27.992913964 +0000 UTC m=+43.677011223 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls") pod "image-registry-97cbdb97b-h88tb" (UID: "278faa32-e840-4525-a31d-cf894fdc4b2d") : secret "image-registry-tls" not found Jul 08 17:39:25.993611 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:25.993347 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 17:39:25.993611 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:25.993395 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls podName:e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:27.993380341 +0000 UTC m=+43.677477601 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls") pod "dns-default-rn2rl" (UID: "e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60") : secret "dns-default-metrics-tls" not found Jul 08 17:39:26.157577 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:26.157507 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-blvkt" event={"ID":"f4e9d179-2384-44e7-afd8-cb22399b829f","Type":"ContainerStarted","Data":"32abe239b6a09b39a8688e410de0ee240cca9a2d3d2bb74af4ebb9fe22ee78e3"} Jul 08 17:39:26.184981 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:26.184924 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-blvkt" podStartSLOduration=8.865206463 podStartE2EDuration="41.184903276s" totalImagesPullingTime="32.319696813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:38:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:39:26.183421761 +0000 UTC m=+41.867519042" watchObservedRunningTime="2026-07-08 17:39:26.184903276 +0000 UTC m=+41.869000557" Jul 08 17:39:26.312517 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:26.312461 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:26.411898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:26.411276 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:26.515899 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:26.515865 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:27.809250 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:27.809215 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-57fr6\" (UID: \"e2388ecf-b94e-4738-9818-4ab78c6d1c80\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" Jul 08 17:39:27.809738 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:27.809353 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 17:39:27.809738 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:27.809362 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:27.809738 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:27.809410 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:27.809738 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:27.809427 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls podName:e2388ecf-b94e-4738-9818-4ab78c6d1c80 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:31.809407039 +0000 UTC m=+47.493504294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-57fr6" (UID: "e2388ecf-b94e-4738-9818-4ab78c6d1c80") : secret "cluster-monitoring-operator-tls" not found Jul 08 17:39:27.809738 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:27.809494 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle podName:7811ee0a-b7e8-45d3-981f-4f81813aeae7 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:31.809480443 +0000 UTC m=+47.493577711 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle") pod "router-default-76b767f9f7-wwf7r" (UID: "7811ee0a-b7e8-45d3-981f-4f81813aeae7") : configmap references non-existent config key: service-ca.crt Jul 08 17:39:27.809738 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:27.809495 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 17:39:27.809738 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:27.809558 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs podName:7811ee0a-b7e8-45d3-981f-4f81813aeae7 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:31.809542567 +0000 UTC m=+47.493639835 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs") pod "router-default-76b767f9f7-wwf7r" (UID: "7811ee0a-b7e8-45d3-981f-4f81813aeae7") : secret "router-metrics-certs-default" not found Jul 08 17:39:27.910597 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:27.910568 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-pg8ww\" (UID: \"9df45c99-7be9-417d-9722-31dfff063d5b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww" Jul 08 17:39:27.910798 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:27.910755 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 17:39:27.910862 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:27.910828 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls podName:9df45c99-7be9-417d-9722-31dfff063d5b nodeName:}" failed. No retries permitted until 2026-07-08 17:39:31.910810597 +0000 UTC m=+47.594907855 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-pg8ww" (UID: "9df45c99-7be9-417d-9722-31dfff063d5b") : secret "samples-operator-tls" not found Jul 08 17:39:28.011964 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:28.011928 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls\") pod \"dns-default-rn2rl\" (UID: \"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60\") " pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:28.012125 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:28.012016 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:28.012125 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:28.012072 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 17:39:28.012217 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:28.012142 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls podName:e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:32.012126486 +0000 UTC m=+47.696223742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls") pod "dns-default-rn2rl" (UID: "e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60") : secret "dns-default-metrics-tls" not found Jul 08 17:39:28.012217 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:28.012147 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 17:39:28.012217 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:28.012162 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97cbdb97b-h88tb: secret "image-registry-tls" not found Jul 08 17:39:28.012347 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:28.012237 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls podName:278faa32-e840-4525-a31d-cf894fdc4b2d nodeName:}" failed. No retries permitted until 2026-07-08 17:39:32.012221887 +0000 UTC m=+47.696319141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls") pod "image-registry-97cbdb97b-h88tb" (UID: "278faa32-e840-4525-a31d-cf894fdc4b2d") : secret "image-registry-tls" not found Jul 08 17:39:30.532291 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:30.532235 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:30.597729 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:30.597692 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:31.093105 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:31.093040 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:31.154361 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:31.154317 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:31.850291 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:31.850231 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:31.850873 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:31.850327 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:31.850873 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:31.850392 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle podName:7811ee0a-b7e8-45d3-981f-4f81813aeae7 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:39.850367972 +0000 UTC m=+55.534465227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle") pod "router-default-76b767f9f7-wwf7r" (UID: "7811ee0a-b7e8-45d3-981f-4f81813aeae7") : configmap references non-existent config key: service-ca.crt Jul 08 17:39:31.850873 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:31.850456 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 17:39:31.850873 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:31.850493 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-57fr6\" (UID: \"e2388ecf-b94e-4738-9818-4ab78c6d1c80\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" Jul 08 17:39:31.850873 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:31.850517 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs podName:7811ee0a-b7e8-45d3-981f-4f81813aeae7 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:39.850501464 +0000 UTC m=+55.534598734 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs") pod "router-default-76b767f9f7-wwf7r" (UID: "7811ee0a-b7e8-45d3-981f-4f81813aeae7") : secret "router-metrics-certs-default" not found Jul 08 17:39:31.850873 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:31.850561 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 17:39:31.850873 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:31.850607 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls podName:e2388ecf-b94e-4738-9818-4ab78c6d1c80 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:39.850595483 +0000 UTC m=+55.534692740 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-57fr6" (UID: "e2388ecf-b94e-4738-9818-4ab78c6d1c80") : secret "cluster-monitoring-operator-tls" not found Jul 08 17:39:31.951707 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:31.951665 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-pg8ww\" (UID: \"9df45c99-7be9-417d-9722-31dfff063d5b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww" Jul 08 17:39:31.951897 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:31.951826 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 17:39:31.951966 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:31.951908 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls podName:9df45c99-7be9-417d-9722-31dfff063d5b nodeName:}" failed. No retries permitted until 2026-07-08 17:39:39.951884922 +0000 UTC m=+55.635982179 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-pg8ww" (UID: "9df45c99-7be9-417d-9722-31dfff063d5b") : secret "samples-operator-tls" not found Jul 08 17:39:32.052422 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:32.052383 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls\") pod \"dns-default-rn2rl\" (UID: \"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60\") " pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:32.052664 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:32.052484 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:32.052664 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:32.052536 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 17:39:32.052664 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:32.052612 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls podName:e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:40.052593596 +0000 UTC m=+55.736690866 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls") pod "dns-default-rn2rl" (UID: "e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60") : secret "dns-default-metrics-tls" not found Jul 08 17:39:32.052664 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:32.052657 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 17:39:32.052820 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:32.052676 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97cbdb97b-h88tb: secret "image-registry-tls" not found Jul 08 17:39:32.052820 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:32.052721 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls podName:278faa32-e840-4525-a31d-cf894fdc4b2d nodeName:}" failed. No retries permitted until 2026-07-08 17:39:40.052708707 +0000 UTC m=+55.736805962 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls") pod "image-registry-97cbdb97b-h88tb" (UID: "278faa32-e840-4525-a31d-cf894fdc4b2d") : secret "image-registry-tls" not found Jul 08 17:39:33.180110 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.179321 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hfbxp" event={"ID":"ce741a8f-ef63-4a40-a144-a4ce45e2e649","Type":"ContainerStarted","Data":"29738f4a724b58f62229c00cc9382877916e3926711ef684278b93087ac629be"} Jul 08 17:39:33.182220 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.181786 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-phwp8" event={"ID":"9d9fbde7-7603-40a6-9fd9-489246aa05bc","Type":"ContainerStarted","Data":"77b0028cedd6b8b5b8427275e96e7a693aac7a5504b9178d500724ff28c5c53f"} Jul 08 17:39:33.183985 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.183594 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tnrxd" event={"ID":"cca60bdc-e638-44e1-a091-9958e5de0c23","Type":"ContainerStarted","Data":"4b731ea40bef0ea146c78bcb187f5844fbd96c048324ab93e1d9fd787c1771ef"} Jul 08 17:39:33.185667 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.185613 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-h6pfl" event={"ID":"d9f0ea1a-0f7d-496d-bdc4-b0cf946ba514","Type":"ContainerStarted","Data":"29b2d70343962a326cfeb47171c6a263f0447d7f619148d747853a9d8736a390"} Jul 08 17:39:33.187342 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.187312 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9hqbp" event={"ID":"8b486c24-34df-4756-81fe-95b0e9577762","Type":"ContainerStarted","Data":"dac5aee4a80d80d3f43eb226f5bb7fabcc8fc16fe7c0df532a6dfcfe3b4b5dc7"} Jul 08 17:39:33.189473 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.189450 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" event={"ID":"05b7e47c-d931-46f1-a21d-fcb853e77030","Type":"ContainerStarted","Data":"505e8bbc1c46c7eeb32b461cb1b6ec3eed311af92cf626d9096bf3133ba5f820"} Jul 08 17:39:33.191159 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.191138 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/0.log" Jul 08 17:39:33.191413 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.191358 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-sbh59" event={"ID":"47839b2a-54b6-4723-bba7-f43ef243c1b1","Type":"ContainerDied","Data":"ceb6ea68fe2ba6fca501ec44f1221b00852da8203d447efc3bc1f9b91d2dcdb3"} Jul 08 17:39:33.191491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.191471 2584 scope.go:117] "RemoveContainer" containerID="ceb6ea68fe2ba6fca501ec44f1221b00852da8203d447efc3bc1f9b91d2dcdb3" Jul 08 17:39:33.191784 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.191175 2584 generic.go:356] "Generic (PLEG): container finished" podID="47839b2a-54b6-4723-bba7-f43ef243c1b1" containerID="ceb6ea68fe2ba6fca501ec44f1221b00852da8203d447efc3bc1f9b91d2dcdb3" exitCode=255 Jul 08 17:39:33.200917 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.199948 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hfbxp" podStartSLOduration=1.42958885 podStartE2EDuration="9.199933025s" totalImagesPullingTime="7.770344175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:39:33.199665644 +0000 UTC m=+48.883762922" watchObservedRunningTime="2026-07-08 17:39:33.199933025 +0000 UTC m=+48.884030302" Jul 08 17:39:33.218362 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.216908 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-h6pfl" podStartSLOduration=4.415907073 podStartE2EDuration="12.216888556s" totalImagesPullingTime="7.800981483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:39:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:39:33.214356873 +0000 UTC m=+48.898454153" watchObservedRunningTime="2026-07-08 17:39:33.216888556 +0000 UTC m=+48.900985835" Jul 08 17:39:33.260847 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.260794 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-phwp8" podStartSLOduration=4.433635086 podStartE2EDuration="12.26077253s" totalImagesPullingTime="7.827137444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:39:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:39:33.233618986 +0000 UTC m=+48.917716264" watchObservedRunningTime="2026-07-08 17:39:33.26077253 +0000 UTC m=+48.944869790" Jul 08 17:39:33.281881 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.281829 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-9hqbp" podStartSLOduration=4.486306417 podStartE2EDuration="12.281808726s" totalImagesPullingTime="7.795502309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:39:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:39:33.261268284 +0000 UTC m=+48.945365560" watchObservedRunningTime="2026-07-08 17:39:33.281808726 +0000 UTC m=+48.965906003" Jul 08 17:39:33.283288 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.283228 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tnrxd" podStartSLOduration=4.440864369 podStartE2EDuration="12.283214272s" totalImagesPullingTime="7.842349903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:39:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:39:33.281806283 +0000 UTC m=+48.965903563" watchObservedRunningTime="2026-07-08 17:39:33.283214272 +0000 UTC m=+48.967311555" Jul 08 17:39:33.307134 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.307087 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" podStartSLOduration=4.451997566 podStartE2EDuration="12.307067368s" totalImagesPullingTime="7.855069802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:39:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:39:33.306560111 +0000 UTC m=+48.990657388" watchObservedRunningTime="2026-07-08 17:39:33.307067368 +0000 UTC m=+48.991164645" Jul 08 17:39:33.338409 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.338336 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:33.405676 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.405642 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:33.895349 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.895313 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:33.960286 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:33.960225 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:34.059300 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.059215 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:34.159644 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.159611 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:34.197869 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.197844 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/1.log" Jul 08 17:39:34.198448 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.198430 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/0.log" Jul 08 17:39:34.198542 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.198481 2584 generic.go:356] "Generic (PLEG): container finished" podID="47839b2a-54b6-4723-bba7-f43ef243c1b1" containerID="42568db069426833af3b9ed345915d99d82ac6577b8bad6a750737266281d743" exitCode=255 Jul 08 17:39:34.199232 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.199208 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-sbh59" event={"ID":"47839b2a-54b6-4723-bba7-f43ef243c1b1","Type":"ContainerDied","Data":"42568db069426833af3b9ed345915d99d82ac6577b8bad6a750737266281d743"} Jul 08 17:39:34.199426 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.199266 2584 scope.go:117] "RemoveContainer" containerID="42568db069426833af3b9ed345915d99d82ac6577b8bad6a750737266281d743" Jul 08 17:39:34.199681 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:34.199661 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-sbh59_openshift-console-operator(47839b2a-54b6-4723-bba7-f43ef243c1b1)\"" pod="openshift-console-operator/console-operator-68b757fb67-sbh59" podUID="47839b2a-54b6-4723-bba7-f43ef243c1b1" Jul 08 17:39:34.200363 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.200212 2584 scope.go:117] "RemoveContainer" containerID="ceb6ea68fe2ba6fca501ec44f1221b00852da8203d447efc3bc1f9b91d2dcdb3" Jul 08 17:39:34.222956 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.222922 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:34.335923 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.335817 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:34.438587 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.438541 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:34.529963 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.529922 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:34.586365 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.586270 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9lmjs"] Jul 08 17:39:34.602950 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.602917 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:34.621328 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.621291 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9lmjs"] Jul 08 17:39:34.621483 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.621340 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-sbh59" Jul 08 17:39:34.621483 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.621359 2584 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-sbh59" Jul 08 17:39:34.621483 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.621412 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:34.624478 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.624453 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 17:39:34.624603 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.624561 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 17:39:34.624797 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.624783 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p2ql9\"" Jul 08 17:39:34.663485 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.663449 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:34.677417 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.677392 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c776g\" (UniqueName: \"kubernetes.io/projected/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-api-access-c776g\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:34.677559 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.677477 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c11cd9b6-1207-4c05-ac75-13396ce41e35-crio-socket\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:34.677559 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.677545 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:34.677633 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.677567 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:34.677633 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.677620 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c11cd9b6-1207-4c05-ac75-13396ce41e35-data-volume\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:34.763831 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.763795 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:34.779012 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.778986 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c776g\" (UniqueName: \"kubernetes.io/projected/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-api-access-c776g\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:34.779163 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.779048 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c11cd9b6-1207-4c05-ac75-13396ce41e35-crio-socket\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:34.779217 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.779170 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:34.779217 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.779203 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:34.779358 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.779267 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c11cd9b6-1207-4c05-ac75-13396ce41e35-crio-socket\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:34.779358 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:34.779303 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 17:39:34.779358 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:34.779347 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 17:39:34.779513 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:34.779353 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-rbac-proxy-cm podName:c11cd9b6-1207-4c05-ac75-13396ce41e35 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:35.279339682 +0000 UTC m=+50.963436941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lmjs" (UID: "c11cd9b6-1207-4c05-ac75-13396ce41e35") : configmap "kube-rbac-proxy" not found Jul 08 17:39:34.779513 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.779432 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c11cd9b6-1207-4c05-ac75-13396ce41e35-data-volume\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:34.779513 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:34.779505 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls podName:c11cd9b6-1207-4c05-ac75-13396ce41e35 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:35.279486519 +0000 UTC m=+50.963583780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lmjs" (UID: "c11cd9b6-1207-4c05-ac75-13396ce41e35") : secret "insights-runtime-extractor-tls" not found Jul 08 17:39:34.779727 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.779710 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c11cd9b6-1207-4c05-ac75-13396ce41e35-data-volume\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:34.791905 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.791882 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c776g\" (UniqueName: \"kubernetes.io/projected/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-api-access-c776g\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:34.864032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:34.863955 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:35.041504 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:35.041467 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:35.143590 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:35.143496 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:35.205069 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:35.205044 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/1.log" Jul 08 17:39:35.205481 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:35.205391 2584 scope.go:117] "RemoveContainer" containerID="42568db069426833af3b9ed345915d99d82ac6577b8bad6a750737266281d743" Jul 08 17:39:35.205552 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:35.205534 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-sbh59_openshift-console-operator(47839b2a-54b6-4723-bba7-f43ef243c1b1)\"" pod="openshift-console-operator/console-operator-68b757fb67-sbh59" podUID="47839b2a-54b6-4723-bba7-f43ef243c1b1" Jul 08 17:39:35.250015 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:35.249975 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:35.285183 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:35.285140 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:35.285350 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:35.285201 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:35.286624 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:35.286601 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 17:39:35.286748 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:35.286684 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls podName:c11cd9b6-1207-4c05-ac75-13396ce41e35 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:36.286662178 +0000 UTC m=+51.970759450 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lmjs" (UID: "c11cd9b6-1207-4c05-ac75-13396ce41e35") : secret "insights-runtime-extractor-tls" not found Jul 08 17:39:35.287282 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:35.287262 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 17:39:35.287388 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:35.287328 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-rbac-proxy-cm podName:c11cd9b6-1207-4c05-ac75-13396ce41e35 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:36.287303362 +0000 UTC m=+51.971400630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lmjs" (UID: "c11cd9b6-1207-4c05-ac75-13396ce41e35") : configmap "kube-rbac-proxy" not found Jul 08 17:39:35.344058 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:35.344020 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:36.208826 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:36.208790 2584 scope.go:117] "RemoveContainer" containerID="42568db069426833af3b9ed345915d99d82ac6577b8bad6a750737266281d743" Jul 08 17:39:36.209182 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:36.208951 2584 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-sbh59_openshift-console-operator(47839b2a-54b6-4723-bba7-f43ef243c1b1)\"" pod="openshift-console-operator/console-operator-68b757fb67-sbh59" podUID="47839b2a-54b6-4723-bba7-f43ef243c1b1" Jul 08 17:39:36.295552 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:36.295518 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:36.295552 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:36.295552 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:36.295729 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:36.295667 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 17:39:36.295777 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:36.295734 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-rbac-proxy-cm podName:c11cd9b6-1207-4c05-ac75-13396ce41e35 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:38.295716244 +0000 UTC m=+53.979813504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lmjs" (UID: "c11cd9b6-1207-4c05-ac75-13396ce41e35") : configmap "kube-rbac-proxy" not found Jul 08 17:39:36.295777 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:36.295676 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 17:39:36.295888 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:36.295864 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls podName:c11cd9b6-1207-4c05-ac75-13396ce41e35 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:38.295837729 +0000 UTC m=+53.979935041 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lmjs" (UID: "c11cd9b6-1207-4c05-ac75-13396ce41e35") : secret "insights-runtime-extractor-tls" not found Jul 08 17:39:36.681032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:36.680978 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:37.199983 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:37.199953 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8h4h6_e8468fb3-5642-43d6-95e5-c524fb532087/dns-node-resolver/0.log" Jul 08 17:39:37.661717 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:37.661675 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:37.999636 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:37.999570 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rhlxp_747083d6-492d-413c-99e7-364aca1c1cee/node-ca/0.log" Jul 08 17:39:38.315115 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:38.315039 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:38.315115 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:38.315074 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:38.315307 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:38.315191 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 17:39:38.315307 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:38.315222 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 17:39:38.315307 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:38.315271 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls podName:c11cd9b6-1207-4c05-ac75-13396ce41e35 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:42.315254278 +0000 UTC m=+57.999351545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lmjs" (UID: "c11cd9b6-1207-4c05-ac75-13396ce41e35") : secret "insights-runtime-extractor-tls" not found Jul 08 17:39:38.315307 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:38.315287 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-rbac-proxy-cm podName:c11cd9b6-1207-4c05-ac75-13396ce41e35 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:42.315280472 +0000 UTC m=+57.999377727 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lmjs" (UID: "c11cd9b6-1207-4c05-ac75-13396ce41e35") : configmap "kube-rbac-proxy" not found Jul 08 17:39:38.799896 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:38.799864 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hfbxp_ce741a8f-ef63-4a40-a144-a4ce45e2e649/serve-healthcheck-canary/0.log" Jul 08 17:39:39.402408 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:39.402372 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-tnrxd_cca60bdc-e638-44e1-a091-9958e5de0c23/kube-storage-version-migrator-operator/0.log" Jul 08 17:39:39.928712 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:39.928676 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:39.928712 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:39.928719 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:39.929116 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:39.928782 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-57fr6\" (UID: \"e2388ecf-b94e-4738-9818-4ab78c6d1c80\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" Jul 08 17:39:39.929116 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:39.928851 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle podName:7811ee0a-b7e8-45d3-981f-4f81813aeae7 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:55.928834646 +0000 UTC m=+71.612931901 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle") pod "router-default-76b767f9f7-wwf7r" (UID: "7811ee0a-b7e8-45d3-981f-4f81813aeae7") : configmap references non-existent config key: service-ca.crt Jul 08 17:39:39.929116 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:39.928886 2584 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 17:39:39.929116 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:39.928892 2584 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 17:39:39.929116 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:39.928922 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls podName:e2388ecf-b94e-4738-9818-4ab78c6d1c80 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:55.928912324 +0000 UTC m=+71.613009580 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-57fr6" (UID: "e2388ecf-b94e-4738-9818-4ab78c6d1c80") : secret "cluster-monitoring-operator-tls" not found Jul 08 17:39:39.929116 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:39.928938 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs podName:7811ee0a-b7e8-45d3-981f-4f81813aeae7 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:55.928928518 +0000 UTC m=+71.613025773 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs") pod "router-default-76b767f9f7-wwf7r" (UID: "7811ee0a-b7e8-45d3-981f-4f81813aeae7") : secret "router-metrics-certs-default" not found Jul 08 17:39:40.029928 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:40.029892 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-pg8ww\" (UID: \"9df45c99-7be9-417d-9722-31dfff063d5b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww" Jul 08 17:39:40.030165 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:40.030039 2584 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 17:39:40.030165 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:40.030112 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls podName:9df45c99-7be9-417d-9722-31dfff063d5b nodeName:}" failed. No retries permitted until 2026-07-08 17:39:56.030095698 +0000 UTC m=+71.714192953 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-pg8ww" (UID: "9df45c99-7be9-417d-9722-31dfff063d5b") : secret "samples-operator-tls" not found Jul 08 17:39:40.064576 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:40.064520 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:40.130623 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:40.130592 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls\") pod \"dns-default-rn2rl\" (UID: \"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60\") " pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:40.130784 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:40.130650 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:40.130784 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:40.130746 2584 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 17:39:40.130847 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:40.130799 2584 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 17:39:40.130847 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:40.130809 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls podName:e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:56.13079242 +0000 UTC m=+71.814889675 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls") pod "dns-default-rn2rl" (UID: "e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60") : secret "dns-default-metrics-tls" not found Jul 08 17:39:40.130847 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:40.130813 2584 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-97cbdb97b-h88tb: secret "image-registry-tls" not found Jul 08 17:39:40.130955 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:40.130866 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls podName:278faa32-e840-4525-a31d-cf894fdc4b2d nodeName:}" failed. No retries permitted until 2026-07-08 17:39:56.130852831 +0000 UTC m=+71.814950090 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls") pod "image-registry-97cbdb97b-h88tb" (UID: "278faa32-e840-4525-a31d-cf894fdc4b2d") : secret "image-registry-tls" not found Jul 08 17:39:40.746368 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:40.746331 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:40.884065 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:40.884032 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:40.987357 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:40.987325 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:41.087035 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:41.086951 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:41.187303 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:41.187262 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:41.294473 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:41.294433 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:41.455527 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:41.455494 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:42.241954 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:42.241916 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:42.352459 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:42.352418 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:42.352459 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:42.352461 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:42.352689 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:42.352567 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 17:39:42.352689 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:42.352599 2584 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 17:39:42.352689 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:42.352630 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls podName:c11cd9b6-1207-4c05-ac75-13396ce41e35 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:50.352615385 +0000 UTC m=+66.036712645 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lmjs" (UID: "c11cd9b6-1207-4c05-ac75-13396ce41e35") : secret "insights-runtime-extractor-tls" not found Jul 08 17:39:42.352689 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:42.352649 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-rbac-proxy-cm podName:c11cd9b6-1207-4c05-ac75-13396ce41e35 nodeName:}" failed. No retries permitted until 2026-07-08 17:39:50.352636569 +0000 UTC m=+66.036733824 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lmjs" (UID: "c11cd9b6-1207-4c05-ac75-13396ce41e35") : configmap "kube-rbac-proxy" not found Jul 08 17:39:42.850871 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:42.850839 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:43.047040 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:43.047004 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:43.447124 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:43.447085 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:43.653812 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:43.653780 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:44.045085 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:44.045049 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:44.400287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:44.400252 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:44.502638 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:44.502603 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:44.652846 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:44.652764 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:44.749037 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:44.748998 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:44.847741 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:44.847709 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:45.247094 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:45.247052 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:45.345574 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:45.345539 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:46.048851 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:46.048812 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:46.149625 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:46.149587 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:46.251087 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:46.251050 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:46.450175 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:46.450135 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:46.644260 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:46.644214 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:46.863302 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:46.863203 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:46.968904 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:46.968862 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:47.075407 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:47.075375 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:47.828533 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:47.828499 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:48.244542 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:48.244509 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:48.445888 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:48.445852 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:48.646969 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:48.646940 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:48.850951 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:48.850913 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:49.043375 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:49.043290 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:49.123139 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:49.123112 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-krx82" Jul 08 17:39:50.043864 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.043825 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:50.245944 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.245911 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:50.417896 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.417860 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:50.418080 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.417903 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:50.418080 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:50.417988 2584 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 17:39:50.418080 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:39:50.418060 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls podName:c11cd9b6-1207-4c05-ac75-13396ce41e35 nodeName:}" failed. No retries permitted until 2026-07-08 17:40:06.418042765 +0000 UTC m=+82.102140020 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lmjs" (UID: "c11cd9b6-1207-4c05-ac75-13396ce41e35") : secret "insights-runtime-extractor-tls" not found Jul 08 17:39:50.418520 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.418499 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c11cd9b6-1207-4c05-ac75-13396ce41e35-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:39:50.453123 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.453086 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:50.619186 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.619155 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4ksh\" (UniqueName: \"kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh\") pod \"network-check-target-d6xks\" (UID: \"135de3f6-fc59-4070-9899-88ac1d14ffde\") " pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:50.622040 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.622003 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4ksh\" (UniqueName: \"kubernetes.io/projected/135de3f6-fc59-4070-9899-88ac1d14ffde-kube-api-access-k4ksh\") pod \"network-check-target-d6xks\" (UID: \"135de3f6-fc59-4070-9899-88ac1d14ffde\") " pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:50.720844 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.720754 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs\") pod \"network-metrics-daemon-h6mc5\" (UID: \"a1fba745-34fb-4f41-b257-c6146b4f88fa\") " pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:39:50.720844 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.720841 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret\") pod \"global-pull-secret-syncer-gbq5c\" (UID: \"ac3c6d61-29c8-4bc9-82d8-78177eb9045c\") " pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:39:50.723621 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.723605 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 17:39:50.724415 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.724400 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 17:39:50.733742 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.733713 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ac3c6d61-29c8-4bc9-82d8-78177eb9045c-original-pull-secret\") pod \"global-pull-secret-syncer-gbq5c\" (UID: \"ac3c6d61-29c8-4bc9-82d8-78177eb9045c\") " pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:39:50.733904 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.733793 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1fba745-34fb-4f41-b257-c6146b4f88fa-metrics-certs\") pod \"network-metrics-daemon-h6mc5\" (UID: \"a1fba745-34fb-4f41-b257-c6146b4f88fa\") " pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:39:50.847477 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.847437 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:50.876194 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.876155 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jwhsf\"" Jul 08 17:39:50.883751 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.883729 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:50.885329 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.885308 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d5skf\"" Jul 08 17:39:50.893271 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.893255 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h6mc5" Jul 08 17:39:50.894397 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:50.894383 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gbq5c" Jul 08 17:39:51.033078 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:51.033004 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-d6xks"] Jul 08 17:39:51.036305 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:39:51.036278 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod135de3f6_fc59_4070_9899_88ac1d14ffde.slice/crio-d7247b8e2563587a43655b5f9ca7a7b962ea3b5b94739fdabeefa7473ed4b7e7 WatchSource:0}: Error finding container d7247b8e2563587a43655b5f9ca7a7b962ea3b5b94739fdabeefa7473ed4b7e7: Status 404 returned error can't find the container with id d7247b8e2563587a43655b5f9ca7a7b962ea3b5b94739fdabeefa7473ed4b7e7 Jul 08 17:39:51.246506 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:51.246421 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:51.256122 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:51.256093 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d6xks" event={"ID":"135de3f6-fc59-4070-9899-88ac1d14ffde","Type":"ContainerStarted","Data":"b322272d3f84c47dcaa7ffd69ede2ba0b04a38c26e497cf472b86965971720ee"} Jul 08 17:39:51.256278 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:51.256138 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-d6xks" event={"ID":"135de3f6-fc59-4070-9899-88ac1d14ffde","Type":"ContainerStarted","Data":"d7247b8e2563587a43655b5f9ca7a7b962ea3b5b94739fdabeefa7473ed4b7e7"} Jul 08 17:39:51.256278 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:51.256215 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:39:51.262341 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:51.262180 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gbq5c"] Jul 08 17:39:51.265056 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:39:51.264944 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3c6d61_29c8_4bc9_82d8_78177eb9045c.slice/crio-cefc4b666bf348acd93c9df9a2e5a8ce6e2f8eb94c34faabba8e8c561585646e WatchSource:0}: Error finding container cefc4b666bf348acd93c9df9a2e5a8ce6e2f8eb94c34faabba8e8c561585646e: Status 404 returned error can't find the container with id cefc4b666bf348acd93c9df9a2e5a8ce6e2f8eb94c34faabba8e8c561585646e Jul 08 17:39:51.265171 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:51.265156 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h6mc5"] Jul 08 17:39:51.267422 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:39:51.267398 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1fba745_34fb_4f41_b257_c6146b4f88fa.slice/crio-c1d6149ff0930f96e9d011d4c0822911f6684b58e6940c2d6ace226c38d308c6 WatchSource:0}: Error finding container c1d6149ff0930f96e9d011d4c0822911f6684b58e6940c2d6ace226c38d308c6: Status 404 returned error can't find the container with id c1d6149ff0930f96e9d011d4c0822911f6684b58e6940c2d6ace226c38d308c6 Jul 08 17:39:51.275969 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:51.275926 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-d6xks" podStartSLOduration=66.275910473 podStartE2EDuration="1m6.275910473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:39:51.274345005 +0000 UTC m=+66.958442282" watchObservedRunningTime="2026-07-08 17:39:51.275910473 +0000 UTC m=+66.960007750" Jul 08 17:39:51.454523 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:51.454484 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:51.951040 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:51.951009 2584 scope.go:117] "RemoveContainer" containerID="42568db069426833af3b9ed345915d99d82ac6577b8bad6a750737266281d743" Jul 08 17:39:52.263303 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:52.263204 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/1.log" Jul 08 17:39:52.263757 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:52.263331 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-sbh59" event={"ID":"47839b2a-54b6-4723-bba7-f43ef243c1b1","Type":"ContainerStarted","Data":"8c12cdbc1049952546dd07ff8d86b6380d42bab940411afe0fea0d3ce949a9a1"} Jul 08 17:39:52.263757 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:52.263635 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-sbh59" Jul 08 17:39:52.264603 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:52.264579 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gbq5c" event={"ID":"ac3c6d61-29c8-4bc9-82d8-78177eb9045c","Type":"ContainerStarted","Data":"cefc4b666bf348acd93c9df9a2e5a8ce6e2f8eb94c34faabba8e8c561585646e"} Jul 08 17:39:52.265854 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:52.265831 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h6mc5" event={"ID":"a1fba745-34fb-4f41-b257-c6146b4f88fa","Type":"ContainerStarted","Data":"c1d6149ff0930f96e9d011d4c0822911f6684b58e6940c2d6ace226c38d308c6"} Jul 08 17:39:52.285825 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:52.284925 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-sbh59" podStartSLOduration=23.485914231 podStartE2EDuration="31.284901325s" totalImagesPullingTime="7.798987094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:39:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:39:52.283988167 +0000 UTC m=+67.968085450" watchObservedRunningTime="2026-07-08 17:39:52.284901325 +0000 UTC m=+67.968998605" Jul 08 17:39:52.750058 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:52.750025 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-sbh59" Jul 08 17:39:52.931874 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:52.931841 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:53.033388 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:53.033355 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:53.092258 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:53.092199 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:53.228362 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:53.228283 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:53.271409 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:53.271373 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h6mc5" event={"ID":"a1fba745-34fb-4f41-b257-c6146b4f88fa","Type":"ContainerStarted","Data":"d007fe73f768636e63d16c4d13635e454071e6b76020e6204b3bfc0ab1be97ca"} Jul 08 17:39:53.271880 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:53.271439 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h6mc5" event={"ID":"a1fba745-34fb-4f41-b257-c6146b4f88fa","Type":"ContainerStarted","Data":"e5b5d27f83efe7d36c5189202110b7749b4d06aaec72ea5ac74434268fddfb34"} Jul 08 17:39:53.292747 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:53.292691 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h6mc5" podStartSLOduration=67.017590838 podStartE2EDuration="1m8.292668326s" totalImagesPullingTime="1.275077488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:39:53.291618519 +0000 UTC m=+68.975715788" watchObservedRunningTime="2026-07-08 17:39:53.292668326 +0000 UTC m=+68.976765604" Jul 08 17:39:53.610694 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:53.610597 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:53.711818 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:53.711784 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:53.812160 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:53.812127 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:53.865812 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:53.865758 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:54.217641 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:54.217550 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:54.275278 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:54.274676 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:54.419711 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:54.419675 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:54.622789 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:54.622756 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:54.774581 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:54.774541 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:55.840573 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:55.840534 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:55.945173 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:55.945134 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:55.965436 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:55.965405 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-57fr6\" (UID: \"e2388ecf-b94e-4738-9818-4ab78c6d1c80\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" Jul 08 17:39:55.965559 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:55.965477 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:55.965559 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:55.965508 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:55.966134 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:55.966112 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7811ee0a-b7e8-45d3-981f-4f81813aeae7-service-ca-bundle\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:55.968012 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:55.967991 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7811ee0a-b7e8-45d3-981f-4f81813aeae7-metrics-certs\") pod \"router-default-76b767f9f7-wwf7r\" (UID: \"7811ee0a-b7e8-45d3-981f-4f81813aeae7\") " pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:55.968095 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:55.968054 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e2388ecf-b94e-4738-9818-4ab78c6d1c80-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-57fr6\" (UID: \"e2388ecf-b94e-4738-9818-4ab78c6d1c80\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" Jul 08 17:39:55.982059 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:55.982030 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8qxg7\"" Jul 08 17:39:55.991031 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:55.991007 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:56.066861 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.066825 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-pg8ww\" (UID: \"9df45c99-7be9-417d-9722-31dfff063d5b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww" Jul 08 17:39:56.069188 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.069168 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9df45c99-7be9-417d-9722-31dfff063d5b-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-pg8ww\" (UID: \"9df45c99-7be9-417d-9722-31dfff063d5b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww" Jul 08 17:39:56.132695 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.132669 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-76b767f9f7-wwf7r"] Jul 08 17:39:56.134474 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.134453 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5676c967b7-2mqgn"] Jul 08 17:39:56.136061 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:39:56.136033 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7811ee0a_b7e8_45d3_981f_4f81813aeae7.slice/crio-609d46e5893429c2c6cd27bec8bcf2d948ff8e6179407a988f66c93c12dcd622 WatchSource:0}: Error finding container 609d46e5893429c2c6cd27bec8bcf2d948ff8e6179407a988f66c93c12dcd622: Status 404 returned error can't find the container with id 609d46e5893429c2c6cd27bec8bcf2d948ff8e6179407a988f66c93c12dcd622 Jul 08 17:39:56.139766 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.139749 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5676c967b7-2mqgn" Jul 08 17:39:56.142111 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.142091 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 08 17:39:56.143445 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.142974 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 08 17:39:56.143445 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.142989 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 08 17:39:56.143445 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.142974 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 08 17:39:56.152750 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.152732 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5676c967b7-2mqgn"] Jul 08 17:39:56.168195 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.168172 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:56.168327 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.168259 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls\") pod \"dns-default-rn2rl\" (UID: \"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60\") " pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:56.170496 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.170470 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60-metrics-tls\") pod \"dns-default-rn2rl\" (UID: \"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60\") " pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:56.170650 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.170594 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls\") pod \"image-registry-97cbdb97b-h88tb\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:56.198283 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.198195 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2bzl9\"" Jul 08 17:39:56.206493 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.206447 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rn2rl" Jul 08 17:39:56.225542 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.225483 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:56.231448 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.230438 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-vtf7x"] Jul 08 17:39:56.233868 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.233847 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-vtf7x" Jul 08 17:39:56.236592 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.236567 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7xhfb\"" Jul 08 17:39:56.236805 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.236773 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 08 17:39:56.237061 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.236836 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 08 17:39:56.244838 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.244811 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-vtf7x"] Jul 08 17:39:56.259450 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.258327 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-64zpd\"" Jul 08 17:39:56.266465 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.266145 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" Jul 08 17:39:56.269127 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.268950 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/01be7efc-bd33-4048-a944-ef4806333b01-klusterlet-config\") pod \"klusterlet-addon-workmgr-5676c967b7-2mqgn\" (UID: \"01be7efc-bd33-4048-a944-ef4806333b01\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5676c967b7-2mqgn" Jul 08 17:39:56.269127 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.269062 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llpw4\" (UniqueName: \"kubernetes.io/projected/01be7efc-bd33-4048-a944-ef4806333b01-kube-api-access-llpw4\") pod \"klusterlet-addon-workmgr-5676c967b7-2mqgn\" (UID: \"01be7efc-bd33-4048-a944-ef4806333b01\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5676c967b7-2mqgn" Jul 08 17:39:56.269127 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.269088 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01be7efc-bd33-4048-a944-ef4806333b01-tmp\") pod \"klusterlet-addon-workmgr-5676c967b7-2mqgn\" (UID: \"01be7efc-bd33-4048-a944-ef4806333b01\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5676c967b7-2mqgn" Jul 08 17:39:56.284633 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.284035 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76b767f9f7-wwf7r" event={"ID":"7811ee0a-b7e8-45d3-981f-4f81813aeae7","Type":"ContainerStarted","Data":"611907e599f78501a32df4343ef9ced62f1a87e437e748afc4b2a3e628630771"} Jul 08 17:39:56.284633 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.284094 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-76b767f9f7-wwf7r" event={"ID":"7811ee0a-b7e8-45d3-981f-4f81813aeae7","Type":"ContainerStarted","Data":"609d46e5893429c2c6cd27bec8bcf2d948ff8e6179407a988f66c93c12dcd622"} Jul 08 17:39:56.298513 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.298466 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:56.306548 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.306510 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-96l64\"" Jul 08 17:39:56.312101 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.312080 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:39:56.313397 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.313016 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-76b767f9f7-wwf7r" podStartSLOduration=35.312995612 podStartE2EDuration="35.312995612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:39:56.311876727 +0000 UTC m=+71.995974005" watchObservedRunningTime="2026-07-08 17:39:56.312995612 +0000 UTC m=+71.997092978" Jul 08 17:39:56.359936 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.359687 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-mkl8h\"" Jul 08 17:39:56.368628 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.368329 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww" Jul 08 17:39:56.371503 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.370600 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/01be7efc-bd33-4048-a944-ef4806333b01-klusterlet-config\") pod \"klusterlet-addon-workmgr-5676c967b7-2mqgn\" (UID: \"01be7efc-bd33-4048-a944-ef4806333b01\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5676c967b7-2mqgn" Jul 08 17:39:56.371503 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.370622 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rn2rl"] Jul 08 17:39:56.371503 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.370653 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg2hk\" (UniqueName: \"kubernetes.io/projected/dc218f8b-426c-4d44-a0b5-09f65c1f2747-kube-api-access-lg2hk\") pod \"downloads-685bfb5f64-vtf7x\" (UID: \"dc218f8b-426c-4d44-a0b5-09f65c1f2747\") " pod="openshift-console/downloads-685bfb5f64-vtf7x" Jul 08 17:39:56.371503 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.370728 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llpw4\" (UniqueName: \"kubernetes.io/projected/01be7efc-bd33-4048-a944-ef4806333b01-kube-api-access-llpw4\") pod \"klusterlet-addon-workmgr-5676c967b7-2mqgn\" (UID: \"01be7efc-bd33-4048-a944-ef4806333b01\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5676c967b7-2mqgn" Jul 08 17:39:56.371503 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.370767 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01be7efc-bd33-4048-a944-ef4806333b01-tmp\") pod \"klusterlet-addon-workmgr-5676c967b7-2mqgn\" (UID: \"01be7efc-bd33-4048-a944-ef4806333b01\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5676c967b7-2mqgn" Jul 08 17:39:56.371503 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.371136 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01be7efc-bd33-4048-a944-ef4806333b01-tmp\") pod \"klusterlet-addon-workmgr-5676c967b7-2mqgn\" (UID: \"01be7efc-bd33-4048-a944-ef4806333b01\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5676c967b7-2mqgn" Jul 08 17:39:56.373843 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.373824 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/01be7efc-bd33-4048-a944-ef4806333b01-klusterlet-config\") pod \"klusterlet-addon-workmgr-5676c967b7-2mqgn\" (UID: \"01be7efc-bd33-4048-a944-ef4806333b01\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5676c967b7-2mqgn" Jul 08 17:39:56.386549 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.386484 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llpw4\" (UniqueName: \"kubernetes.io/projected/01be7efc-bd33-4048-a944-ef4806333b01-kube-api-access-llpw4\") pod \"klusterlet-addon-workmgr-5676c967b7-2mqgn\" (UID: \"01be7efc-bd33-4048-a944-ef4806333b01\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5676c967b7-2mqgn" Jul 08 17:39:56.400675 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.400601 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:56.431990 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.431899 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6"] Jul 08 17:39:56.450091 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.450063 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5676c967b7-2mqgn" Jul 08 17:39:56.471803 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.471752 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg2hk\" (UniqueName: \"kubernetes.io/projected/dc218f8b-426c-4d44-a0b5-09f65c1f2747-kube-api-access-lg2hk\") pod \"downloads-685bfb5f64-vtf7x\" (UID: \"dc218f8b-426c-4d44-a0b5-09f65c1f2747\") " pod="openshift-console/downloads-685bfb5f64-vtf7x" Jul 08 17:39:56.491070 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.491043 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg2hk\" (UniqueName: \"kubernetes.io/projected/dc218f8b-426c-4d44-a0b5-09f65c1f2747-kube-api-access-lg2hk\") pod \"downloads-685bfb5f64-vtf7x\" (UID: \"dc218f8b-426c-4d44-a0b5-09f65c1f2747\") " pod="openshift-console/downloads-685bfb5f64-vtf7x" Jul 08 17:39:56.498792 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.497015 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-97cbdb97b-h88tb"] Jul 08 17:39:56.500746 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.500708 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:56.546312 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.546282 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-vtf7x" Jul 08 17:39:56.592628 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.592583 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:39:56.991305 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.991226 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:56.994034 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:56.994011 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:57.287810 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:57.287710 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:57.289106 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:57.289084 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-76b767f9f7-wwf7r" Jul 08 17:39:57.779382 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:39:57.779351 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2388ecf_b94e_4738_9818_4ab78c6d1c80.slice/crio-dd39d40b6bcb5fdbbc134a667c9950dc108db4a1cf0baf6e6b774371648f7bb5 WatchSource:0}: Error finding container dd39d40b6bcb5fdbbc134a667c9950dc108db4a1cf0baf6e6b774371648f7bb5: Status 404 returned error can't find the container with id dd39d40b6bcb5fdbbc134a667c9950dc108db4a1cf0baf6e6b774371648f7bb5 Jul 08 17:39:57.779969 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:39:57.779936 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278faa32_e840_4525_a31d_cf894fdc4b2d.slice/crio-eb648d35f9d88853b90d2f1a52c1633b4fa121cf639db7c148dc147ba5bb65d4 WatchSource:0}: Error finding container eb648d35f9d88853b90d2f1a52c1633b4fa121cf639db7c148dc147ba5bb65d4: Status 404 returned error can't find the container with id eb648d35f9d88853b90d2f1a52c1633b4fa121cf639db7c148dc147ba5bb65d4 Jul 08 17:39:57.781322 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:39:57.781295 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f6ff84_c9d3_4d4a_9c7f_7eda26065d60.slice/crio-3a12de1cf8a535406c171d3fc5818591c2247c35ba368edcb0bb468eb38fefa6 WatchSource:0}: Error finding container 3a12de1cf8a535406c171d3fc5818591c2247c35ba368edcb0bb468eb38fefa6: Status 404 returned error can't find the container with id 3a12de1cf8a535406c171d3fc5818591c2247c35ba368edcb0bb468eb38fefa6 Jul 08 17:39:57.963525 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:57.963451 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww"] Jul 08 17:39:57.987895 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:57.987807 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-vtf7x"] Jul 08 17:39:57.999425 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:39:57.999393 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc218f8b_426c_4d44_a0b5_09f65c1f2747.slice/crio-ad52a3505d4ff60becf5119e1f99ae0cf280fd6b11c64eed656fc3e5ba8f8cdb WatchSource:0}: Error finding container ad52a3505d4ff60becf5119e1f99ae0cf280fd6b11c64eed656fc3e5ba8f8cdb: Status 404 returned error can't find the container with id ad52a3505d4ff60becf5119e1f99ae0cf280fd6b11c64eed656fc3e5ba8f8cdb Jul 08 17:39:58.009020 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:58.008954 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5676c967b7-2mqgn"] Jul 08 17:39:58.011750 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:39:58.011651 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01be7efc_bd33_4048_a944_ef4806333b01.slice/crio-a3cf382360df80a095482d47fb4f3358cbacd340da32fbde947ea7db7a4cedf5 WatchSource:0}: Error finding container a3cf382360df80a095482d47fb4f3358cbacd340da32fbde947ea7db7a4cedf5: Status 404 returned error can't find the container with id a3cf382360df80a095482d47fb4f3358cbacd340da32fbde947ea7db7a4cedf5 Jul 08 17:39:58.293472 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:58.293434 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww" event={"ID":"9df45c99-7be9-417d-9722-31dfff063d5b","Type":"ContainerStarted","Data":"5e2e55fc537192fd8398a80d4546d934aa4214f3a980d16d410233c15de01937"} Jul 08 17:39:58.295475 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:58.295444 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gbq5c" event={"ID":"ac3c6d61-29c8-4bc9-82d8-78177eb9045c","Type":"ContainerStarted","Data":"b0b4677fa74d479704e648e916d9090ea8e4e87f675b2d76c391a2cb36c28026"} Jul 08 17:39:58.296993 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:58.296962 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" event={"ID":"e2388ecf-b94e-4738-9818-4ab78c6d1c80","Type":"ContainerStarted","Data":"dd39d40b6bcb5fdbbc134a667c9950dc108db4a1cf0baf6e6b774371648f7bb5"} Jul 08 17:39:58.298375 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:58.298348 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rn2rl" event={"ID":"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60","Type":"ContainerStarted","Data":"3a12de1cf8a535406c171d3fc5818591c2247c35ba368edcb0bb468eb38fefa6"} Jul 08 17:39:58.299618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:58.299593 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-vtf7x" event={"ID":"dc218f8b-426c-4d44-a0b5-09f65c1f2747","Type":"ContainerStarted","Data":"ad52a3505d4ff60becf5119e1f99ae0cf280fd6b11c64eed656fc3e5ba8f8cdb"} Jul 08 17:39:58.301099 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:58.301072 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5676c967b7-2mqgn" event={"ID":"01be7efc-bd33-4048-a944-ef4806333b01","Type":"ContainerStarted","Data":"a3cf382360df80a095482d47fb4f3358cbacd340da32fbde947ea7db7a4cedf5"} Jul 08 17:39:58.302863 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:58.302839 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" event={"ID":"278faa32-e840-4525-a31d-cf894fdc4b2d","Type":"ContainerStarted","Data":"fce80f209f57745681aec76481e0c549935852d891aa30928bf4d089294e8760"} Jul 08 17:39:58.302969 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:58.302882 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" event={"ID":"278faa32-e840-4525-a31d-cf894fdc4b2d","Type":"ContainerStarted","Data":"eb648d35f9d88853b90d2f1a52c1633b4fa121cf639db7c148dc147ba5bb65d4"} Jul 08 17:39:58.314155 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:58.314094 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gbq5c" podStartSLOduration=66.723530912 podStartE2EDuration="1m13.314077826s" totalImagesPullingTime="6.590546914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:39:58.31310549 +0000 UTC m=+73.997202768" watchObservedRunningTime="2026-07-08 17:39:58.314077826 +0000 UTC m=+73.998175103" Jul 08 17:39:58.333011 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:58.332965 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" podStartSLOduration=73.332950365 podStartE2EDuration="1m13.332950365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:39:58.332629285 +0000 UTC m=+74.016726562" watchObservedRunningTime="2026-07-08 17:39:58.332950365 +0000 UTC m=+74.017047643" Jul 08 17:39:59.309264 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:39:59.309182 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:40:00.518491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:00.518309 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:00.585908 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:00.585846 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:02.523583 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:02.523541 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:02.577300 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:02.577258 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:04.734744 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:04.734665 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:04.838729 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:04.838688 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:04.972877 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:04.972837 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:05.201411 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:05.201309 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:05.335358 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:05.335313 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" event={"ID":"e2388ecf-b94e-4738-9818-4ab78c6d1c80","Type":"ContainerStarted","Data":"218e52fe94b8d388510920ab7de63d781fd0ff6697caec15267f73ca55fb2b5d"} Jul 08 17:40:05.337932 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:05.337900 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rn2rl" event={"ID":"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60","Type":"ContainerStarted","Data":"da13983916651a055a73ebce69098046dd90b951f2594a24c0fbc530744f9098"} Jul 08 17:40:05.338095 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:05.337959 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rn2rl" event={"ID":"e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60","Type":"ContainerStarted","Data":"921f5fec6223903988bdf5fd87c8ebae98236652a0ff9f45c6cc5da0095829b8"} Jul 08 17:40:05.338095 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:05.338018 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rn2rl" Jul 08 17:40:05.339944 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:05.339601 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5676c967b7-2mqgn" event={"ID":"01be7efc-bd33-4048-a944-ef4806333b01","Type":"ContainerStarted","Data":"f53cceff14241aada06fd23259e544aff082b8cdf0b48ecb305876244f17a933"} Jul 08 17:40:05.339944 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:05.339733 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5676c967b7-2mqgn" Jul 08 17:40:05.341853 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:05.341824 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5676c967b7-2mqgn" Jul 08 17:40:05.342329 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:05.342298 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww" event={"ID":"9df45c99-7be9-417d-9722-31dfff063d5b","Type":"ContainerStarted","Data":"2f47359b77251f94388fcedd4bdf7a8c7465b5b7b7c58cba90d25b22850b5611"} Jul 08 17:40:05.342457 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:05.342343 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww" event={"ID":"9df45c99-7be9-417d-9722-31dfff063d5b","Type":"ContainerStarted","Data":"eef1e522b5100d0e06a4ecc7bdf3480d1a5d24af81285601a9df0e62aa4a6f6e"} Jul 08 17:40:05.354361 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:05.354296 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-57fr6" podStartSLOduration=37.849147321 podStartE2EDuration="44.354279176s" totalImagesPullingTime="6.505131855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:39:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:40:05.352961113 +0000 UTC m=+81.037058391" watchObservedRunningTime="2026-07-08 17:40:05.354279176 +0000 UTC m=+81.038376452" Jul 08 17:40:05.369870 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:05.369817 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-pg8ww" podStartSLOduration=38.115026035 podStartE2EDuration="44.369797367s" totalImagesPullingTime="6.254771332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:39:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:40:05.369531112 +0000 UTC m=+81.053628388" watchObservedRunningTime="2026-07-08 17:40:05.369797367 +0000 UTC m=+81.053894645" Jul 08 17:40:05.388547 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:05.388497 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rn2rl" podStartSLOduration=34.888969412 podStartE2EDuration="41.388480343s" totalImagesPullingTime="6.499510931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:40:05.387393744 +0000 UTC m=+81.071491020" watchObservedRunningTime="2026-07-08 17:40:05.388480343 +0000 UTC m=+81.072577621" Jul 08 17:40:05.406988 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:05.406934 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5676c967b7-2mqgn" podStartSLOduration=2.776487005 podStartE2EDuration="9.406916298s" totalImagesPullingTime="6.630429293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:40:05.405643189 +0000 UTC m=+81.089740612" watchObservedRunningTime="2026-07-08 17:40:05.406916298 +0000 UTC m=+81.091013574" Jul 08 17:40:05.983449 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:05.983417 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:06.040132 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:06.040086 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:06.109256 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:06.109211 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:06.166908 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:06.166866 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:06.221672 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:06.221634 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:06.466701 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:06.466667 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:40:06.469208 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:06.469180 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c11cd9b6-1207-4c05-ac75-13396ce41e35-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lmjs\" (UID: \"c11cd9b6-1207-4c05-ac75-13396ce41e35\") " pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:40:06.733477 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:06.733405 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-p2ql9\"" Jul 08 17:40:06.741550 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:06.741505 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9lmjs" Jul 08 17:40:06.884196 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:06.884157 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:06.893140 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:06.893111 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9lmjs"] Jul 08 17:40:06.895863 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:40:06.895834 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11cd9b6_1207_4c05_ac75_13396ce41e35.slice/crio-fba1fa9ae5826e7f740178b5138f03e73640ad99a23eb971c00d5a803a86bcf6 WatchSource:0}: Error finding container fba1fa9ae5826e7f740178b5138f03e73640ad99a23eb971c00d5a803a86bcf6: Status 404 returned error can't find the container with id fba1fa9ae5826e7f740178b5138f03e73640ad99a23eb971c00d5a803a86bcf6 Jul 08 17:40:07.351932 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:07.351791 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9lmjs" event={"ID":"c11cd9b6-1207-4c05-ac75-13396ce41e35","Type":"ContainerStarted","Data":"fba1fa9ae5826e7f740178b5138f03e73640ad99a23eb971c00d5a803a86bcf6"} Jul 08 17:40:08.850842 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:08.850800 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:08.914305 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:08.914264 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:09.845758 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:09.845717 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:09.901371 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:09.901310 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:13.009072 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:13.009026 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:13.071897 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:13.071840 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:13.168941 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:13.168901 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:13.492746 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:13.492706 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:13.592852 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:13.592813 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:13.710454 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:13.710368 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:13.775107 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:13.775008 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:15.335063 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:15.335028 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:15.348619 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:15.348592 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rn2rl" Jul 08 17:40:15.542865 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:15.542822 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:15.646399 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:15.646369 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:16.317500 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:16.317457 2584 patch_prober.go:28] interesting pod/image-registry-97cbdb97b-h88tb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 08 17:40:16.317682 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:16.317523 2584 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" podUID="278faa32-e840-4525-a31d-cf894fdc4b2d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 17:40:18.157028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.156977 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:18.413535 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.413460 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:18.503929 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.503843 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:18.529620 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.529586 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wnhc8"] Jul 08 17:40:18.543709 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.543686 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.546197 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.546176 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 17:40:18.546433 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.546417 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cccpp\"" Jul 08 17:40:18.546513 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.546461 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 17:40:18.546985 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.546968 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 17:40:18.547039 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.546968 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 17:40:18.581743 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.581706 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:18.675912 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.675888 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-sys\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.676038 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.675926 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-node-exporter-accelerators-collector-config\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.676038 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.675956 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-node-exporter-textfile\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.676038 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.675987 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-node-exporter-tls\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.676038 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.676010 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltncx\" (UniqueName: \"kubernetes.io/projected/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-kube-api-access-ltncx\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.676263 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.676092 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-node-exporter-wtmp\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.676263 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.676133 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.676263 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.676190 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-root\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.676386 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.676236 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-metrics-client-ca\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.678433 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.678401 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:18.776851 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.776820 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-node-exporter-wtmp\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.776851 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.776863 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.777142 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.776888 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-root\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.777142 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.776928 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-metrics-client-ca\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.777142 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.776968 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-sys\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.777142 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.776991 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-node-exporter-accelerators-collector-config\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.777142 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.777002 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-node-exporter-wtmp\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.777142 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.777019 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-node-exporter-textfile\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.777142 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.777002 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-root\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.777142 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.777085 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-node-exporter-tls\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.777548 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:40:18.777165 2584 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 08 17:40:18.777548 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:40:18.777236 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-node-exporter-tls podName:a1e0969a-1dab-4d4d-b4b2-f3d30910bc69 nodeName:}" failed. No retries permitted until 2026-07-08 17:40:19.277203764 +0000 UTC m=+94.961301023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-node-exporter-tls") pod "node-exporter-wnhc8" (UID: "a1e0969a-1dab-4d4d-b4b2-f3d30910bc69") : secret "node-exporter-tls" not found Jul 08 17:40:18.777548 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.777354 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-sys\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.777548 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.777460 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-node-exporter-textfile\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.777548 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.777531 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltncx\" (UniqueName: \"kubernetes.io/projected/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-kube-api-access-ltncx\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.777784 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.777628 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-metrics-client-ca\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.777834 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.777807 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-node-exporter-accelerators-collector-config\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.778778 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.778743 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:18.779466 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.779448 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.786377 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.786357 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltncx\" (UniqueName: \"kubernetes.io/projected/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-kube-api-access-ltncx\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:18.852935 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.852902 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-97cbdb97b-h88tb"] Jul 08 17:40:18.858306 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.858263 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:40:18.880360 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:18.880326 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:19.127178 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:19.127131 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:19.227166 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:19.227115 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:19.282013 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:19.281973 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-node-exporter-tls\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:19.284826 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:19.284798 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a1e0969a-1dab-4d4d-b4b2-f3d30910bc69-node-exporter-tls\") pod \"node-exporter-wnhc8\" (UID: \"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69\") " pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:19.328523 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:19.328486 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:19.406599 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:19.405980 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9lmjs" event={"ID":"c11cd9b6-1207-4c05-ac75-13396ce41e35","Type":"ContainerStarted","Data":"000c57c5c04340e787bf16ffc54887703f7bf103e322ccdc59efd3bb5e61c82c"} Jul 08 17:40:19.410076 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:19.407841 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-vtf7x" event={"ID":"dc218f8b-426c-4d44-a0b5-09f65c1f2747","Type":"ContainerStarted","Data":"3309a9c3026952a6c899d01b76bbad186d3415ff69a6ee3b824bdcabd7ad2eac"} Jul 08 17:40:19.410076 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:19.408102 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-vtf7x" Jul 08 17:40:19.424798 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:19.424312 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-vtf7x" Jul 08 17:40:19.429149 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:19.428727 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-vtf7x" podStartSLOduration=2.77957642 podStartE2EDuration="23.428707716s" totalImagesPullingTime="20.649131296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:40:19.426621965 +0000 UTC m=+95.110719244" watchObservedRunningTime="2026-07-08 17:40:19.428707716 +0000 UTC m=+95.112804995" Jul 08 17:40:19.450545 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:19.450512 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:19.454327 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:19.454155 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wnhc8" Jul 08 17:40:19.492591 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:40:19.492553 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e0969a_1dab_4d4d_b4b2_f3d30910bc69.slice/crio-36e8299d9b99729686241c24616ebaf13e3540595961235aa855c018670491e9 WatchSource:0}: Error finding container 36e8299d9b99729686241c24616ebaf13e3540595961235aa855c018670491e9: Status 404 returned error can't find the container with id 36e8299d9b99729686241c24616ebaf13e3540595961235aa855c018670491e9 Jul 08 17:40:19.542104 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:19.542068 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:19.939197 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:19.939150 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:20.413880 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:20.413827 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wnhc8" event={"ID":"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69","Type":"ContainerStarted","Data":"36e8299d9b99729686241c24616ebaf13e3540595961235aa855c018670491e9"} Jul 08 17:40:20.416011 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:20.415975 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9lmjs" event={"ID":"c11cd9b6-1207-4c05-ac75-13396ce41e35","Type":"ContainerStarted","Data":"fd6c7492f3f97b439e91d8de317db312e940f0e5b84f9eb7453ab335a4ee50ac"} Jul 08 17:40:21.427871 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:21.427624 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:21.508915 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:21.508838 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:21.608576 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:21.608497 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:21.708758 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:21.708714 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:22.269437 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:22.269409 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-d6xks" Jul 08 17:40:22.430640 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:22.430598 2584 generic.go:356] "Generic (PLEG): container finished" podID="a1e0969a-1dab-4d4d-b4b2-f3d30910bc69" containerID="7ac1ce70078c5fced6167c1faa686eea6a3d891fb32ce61bbd19f26e45b99f1a" exitCode=0 Jul 08 17:40:22.431096 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:22.430727 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wnhc8" event={"ID":"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69","Type":"ContainerDied","Data":"7ac1ce70078c5fced6167c1faa686eea6a3d891fb32ce61bbd19f26e45b99f1a"} Jul 08 17:40:23.201405 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:23.201319 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:23.265977 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:23.265949 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:23.437003 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:23.436921 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wnhc8" event={"ID":"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69","Type":"ContainerStarted","Data":"544cbf028a020343ad24f98108216085d64e869160aaf33f38058815019dd7ef"} Jul 08 17:40:24.035999 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:24.035914 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:24.102861 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:24.102820 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:24.443828 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:24.443781 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9lmjs" event={"ID":"c11cd9b6-1207-4c05-ac75-13396ce41e35","Type":"ContainerStarted","Data":"beb862a29fe0af8d26c457526b5ab567e28cc78f55049de150fc7ab8bba42bc1"} Jul 08 17:40:24.446347 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:24.446313 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wnhc8" event={"ID":"a1e0969a-1dab-4d4d-b4b2-f3d30910bc69","Type":"ContainerStarted","Data":"36f35470103fa25ba18211ff07465aa59adf4ec2601d5c1bcd71d84d2c4bbf5b"} Jul 08 17:40:24.464711 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:24.464651 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9lmjs" podStartSLOduration=33.976445096 podStartE2EDuration="50.464634109s" totalImagesPullingTime="16.488189013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:39:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:40:24.463468139 +0000 UTC m=+100.147565429" watchObservedRunningTime="2026-07-08 17:40:24.464634109 +0000 UTC m=+100.148731396" Jul 08 17:40:24.492333 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:24.492274 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wnhc8" podStartSLOduration=4.512662538 podStartE2EDuration="6.492254218s" totalImagesPullingTime="1.877675334s" totalInitContainerRuntime="101.916346ms" isStatefulPod=true podCreationTimestamp="2026-07-08 17:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:40:24.490840714 +0000 UTC m=+100.174938025" watchObservedRunningTime="2026-07-08 17:40:24.492254218 +0000 UTC m=+100.176351487" Jul 08 17:40:24.529471 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:24.529422 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:24.728601 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:24.728519 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:24.811816 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:24.811762 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:27.145285 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:27.145221 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:27.208909 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:27.208864 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:28.065674 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:28.065639 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:28.135118 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:28.135084 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:29.025704 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:29.025659 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:29.381965 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:29.381929 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:29.477300 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:29.477264 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:30.240885 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:30.240840 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:30.343487 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:30.343450 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:33.517027 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:33.516991 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:33.617979 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:33.617944 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:36.891540 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:36.891502 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:36.990908 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:36.990875 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:37.854476 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:37.854439 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:38.177435 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:38.177403 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:38.285101 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:38.285062 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:38.387793 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:38.387759 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:38.502850 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:38.502762 2584 generic.go:356] "Generic (PLEG): container finished" podID="9d9fbde7-7603-40a6-9fd9-489246aa05bc" containerID="77b0028cedd6b8b5b8427275e96e7a693aac7a5504b9178d500724ff28c5c53f" exitCode=0 Jul 08 17:40:38.502850 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:38.502835 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-phwp8" event={"ID":"9d9fbde7-7603-40a6-9fd9-489246aa05bc","Type":"ContainerDied","Data":"77b0028cedd6b8b5b8427275e96e7a693aac7a5504b9178d500724ff28c5c53f"} Jul 08 17:40:38.503187 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:38.503168 2584 scope.go:117] "RemoveContainer" containerID="77b0028cedd6b8b5b8427275e96e7a693aac7a5504b9178d500724ff28c5c53f" Jul 08 17:40:39.508647 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:39.508616 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-phwp8" event={"ID":"9d9fbde7-7603-40a6-9fd9-489246aa05bc","Type":"ContainerStarted","Data":"c92f6bf177eb379745c1fedadf7ae3b91ab4c1b213310b75859301e41533efd7"} Jul 08 17:40:41.514624 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:41.514584 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:41.614890 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:41.614856 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:41.716888 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:41.716850 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:43.728313 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:43.728276 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:43.828134 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:43.828099 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:43.877693 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:43.877647 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" podUID="278faa32-e840-4525-a31d-cf894fdc4b2d" containerName="registry" containerID="cri-o://fce80f209f57745681aec76481e0c549935852d891aa30928bf4d089294e8760" gracePeriod=30 Jul 08 17:40:44.159445 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.159408 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:40:44.289433 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.289358 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/278faa32-e840-4525-a31d-cf894fdc4b2d-ca-trust-extracted\") pod \"278faa32-e840-4525-a31d-cf894fdc4b2d\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " Jul 08 17:40:44.289433 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.289410 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/278faa32-e840-4525-a31d-cf894fdc4b2d-image-registry-private-configuration\") pod \"278faa32-e840-4525-a31d-cf894fdc4b2d\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " Jul 08 17:40:44.289433 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.289432 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/278faa32-e840-4525-a31d-cf894fdc4b2d-installation-pull-secrets\") pod \"278faa32-e840-4525-a31d-cf894fdc4b2d\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " Jul 08 17:40:44.289690 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.289456 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-certificates\") pod \"278faa32-e840-4525-a31d-cf894fdc4b2d\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " Jul 08 17:40:44.289690 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.289581 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls\") pod \"278faa32-e840-4525-a31d-cf894fdc4b2d\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " Jul 08 17:40:44.289690 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.289631 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-bound-sa-token\") pod \"278faa32-e840-4525-a31d-cf894fdc4b2d\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " Jul 08 17:40:44.289882 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.289711 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhmr7\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-kube-api-access-mhmr7\") pod \"278faa32-e840-4525-a31d-cf894fdc4b2d\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " Jul 08 17:40:44.289882 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.289764 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/278faa32-e840-4525-a31d-cf894fdc4b2d-trusted-ca\") pod \"278faa32-e840-4525-a31d-cf894fdc4b2d\" (UID: \"278faa32-e840-4525-a31d-cf894fdc4b2d\") " Jul 08 17:40:44.289979 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.289909 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "278faa32-e840-4525-a31d-cf894fdc4b2d" (UID: "278faa32-e840-4525-a31d-cf894fdc4b2d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:40:44.290039 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.290025 2584 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-certificates\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:40:44.290365 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.290338 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/278faa32-e840-4525-a31d-cf894fdc4b2d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "278faa32-e840-4525-a31d-cf894fdc4b2d" (UID: "278faa32-e840-4525-a31d-cf894fdc4b2d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:40:44.292413 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.292381 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "278faa32-e840-4525-a31d-cf894fdc4b2d" (UID: "278faa32-e840-4525-a31d-cf894fdc4b2d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:40:44.292642 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.292601 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/278faa32-e840-4525-a31d-cf894fdc4b2d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "278faa32-e840-4525-a31d-cf894fdc4b2d" (UID: "278faa32-e840-4525-a31d-cf894fdc4b2d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:40:44.292757 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.292659 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/278faa32-e840-4525-a31d-cf894fdc4b2d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "278faa32-e840-4525-a31d-cf894fdc4b2d" (UID: "278faa32-e840-4525-a31d-cf894fdc4b2d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:40:44.292757 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.292707 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-kube-api-access-mhmr7" (OuterVolumeSpecName: "kube-api-access-mhmr7") pod "278faa32-e840-4525-a31d-cf894fdc4b2d" (UID: "278faa32-e840-4525-a31d-cf894fdc4b2d"). InnerVolumeSpecName "kube-api-access-mhmr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:40:44.292869 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.292768 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "278faa32-e840-4525-a31d-cf894fdc4b2d" (UID: "278faa32-e840-4525-a31d-cf894fdc4b2d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:40:44.298775 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.298746 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278faa32-e840-4525-a31d-cf894fdc4b2d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "278faa32-e840-4525-a31d-cf894fdc4b2d" (UID: "278faa32-e840-4525-a31d-cf894fdc4b2d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:40:44.390978 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.390935 2584 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/278faa32-e840-4525-a31d-cf894fdc4b2d-trusted-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:40:44.390978 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.390968 2584 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/278faa32-e840-4525-a31d-cf894fdc4b2d-ca-trust-extracted\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:40:44.390978 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.390980 2584 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/278faa32-e840-4525-a31d-cf894fdc4b2d-image-registry-private-configuration\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:40:44.390978 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.390989 2584 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/278faa32-e840-4525-a31d-cf894fdc4b2d-installation-pull-secrets\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:40:44.391274 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.391000 2584 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-registry-tls\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:40:44.391274 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.391008 2584 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-bound-sa-token\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:40:44.391274 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.391015 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhmr7\" (UniqueName: \"kubernetes.io/projected/278faa32-e840-4525-a31d-cf894fdc4b2d-kube-api-access-mhmr7\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:40:44.527386 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.527343 2584 generic.go:356] "Generic (PLEG): container finished" podID="278faa32-e840-4525-a31d-cf894fdc4b2d" containerID="fce80f209f57745681aec76481e0c549935852d891aa30928bf4d089294e8760" exitCode=0 Jul 08 17:40:44.527573 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.527413 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" Jul 08 17:40:44.527573 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.527427 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" event={"ID":"278faa32-e840-4525-a31d-cf894fdc4b2d","Type":"ContainerDied","Data":"fce80f209f57745681aec76481e0c549935852d891aa30928bf4d089294e8760"} Jul 08 17:40:44.527573 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.527507 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-97cbdb97b-h88tb" event={"ID":"278faa32-e840-4525-a31d-cf894fdc4b2d","Type":"ContainerDied","Data":"eb648d35f9d88853b90d2f1a52c1633b4fa121cf639db7c148dc147ba5bb65d4"} Jul 08 17:40:44.527573 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.527539 2584 scope.go:117] "RemoveContainer" containerID="fce80f209f57745681aec76481e0c549935852d891aa30928bf4d089294e8760" Jul 08 17:40:44.535884 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.535864 2584 scope.go:117] "RemoveContainer" containerID="fce80f209f57745681aec76481e0c549935852d891aa30928bf4d089294e8760" Jul 08 17:40:44.536171 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:40:44.536151 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce80f209f57745681aec76481e0c549935852d891aa30928bf4d089294e8760\": container with ID starting with fce80f209f57745681aec76481e0c549935852d891aa30928bf4d089294e8760 not found: ID does not exist" containerID="fce80f209f57745681aec76481e0c549935852d891aa30928bf4d089294e8760" Jul 08 17:40:44.536215 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.536180 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce80f209f57745681aec76481e0c549935852d891aa30928bf4d089294e8760"} err="failed to get container status \"fce80f209f57745681aec76481e0c549935852d891aa30928bf4d089294e8760\": rpc error: code = NotFound desc = could not find container \"fce80f209f57745681aec76481e0c549935852d891aa30928bf4d089294e8760\": container with ID starting with fce80f209f57745681aec76481e0c549935852d891aa30928bf4d089294e8760 not found: ID does not exist" Jul 08 17:40:44.555975 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.555902 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-97cbdb97b-h88tb"] Jul 08 17:40:44.565671 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.565643 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-97cbdb97b-h88tb"] Jul 08 17:40:44.955557 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:44.955523 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="278faa32-e840-4525-a31d-cf894fdc4b2d" path="/var/lib/kubelet/pods/278faa32-e840-4525-a31d-cf894fdc4b2d/volumes" Jul 08 17:40:49.633440 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:49.633402 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:40:59.581048 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:59.581014 2584 generic.go:356] "Generic (PLEG): container finished" podID="05b7e47c-d931-46f1-a21d-fcb853e77030" containerID="505e8bbc1c46c7eeb32b461cb1b6ec3eed311af92cf626d9096bf3133ba5f820" exitCode=0 Jul 08 17:40:59.581493 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:59.581088 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" event={"ID":"05b7e47c-d931-46f1-a21d-fcb853e77030","Type":"ContainerDied","Data":"505e8bbc1c46c7eeb32b461cb1b6ec3eed311af92cf626d9096bf3133ba5f820"} Jul 08 17:40:59.581493 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:40:59.581451 2584 scope.go:117] "RemoveContainer" containerID="505e8bbc1c46c7eeb32b461cb1b6ec3eed311af92cf626d9096bf3133ba5f820" Jul 08 17:41:00.586216 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:41:00.586180 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-fqtgz" event={"ID":"05b7e47c-d931-46f1-a21d-fcb853e77030","Type":"ContainerStarted","Data":"88e6e276a937278a2d8987619001e0f1352028b27d7dae20ae6b39efe11505d6"} Jul 08 17:41:00.614750 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:41:00.614701 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:00.672300 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:41:00.672260 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:04.601357 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:41:04.601325 2584 generic.go:356] "Generic (PLEG): container finished" podID="cca60bdc-e638-44e1-a091-9958e5de0c23" containerID="4b731ea40bef0ea146c78bcb187f5844fbd96c048324ab93e1d9fd787c1771ef" exitCode=0 Jul 08 17:41:04.601772 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:41:04.601388 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tnrxd" event={"ID":"cca60bdc-e638-44e1-a091-9958e5de0c23","Type":"ContainerDied","Data":"4b731ea40bef0ea146c78bcb187f5844fbd96c048324ab93e1d9fd787c1771ef"} Jul 08 17:41:04.601772 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:41:04.601726 2584 scope.go:117] "RemoveContainer" containerID="4b731ea40bef0ea146c78bcb187f5844fbd96c048324ab93e1d9fd787c1771ef" Jul 08 17:41:05.606592 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:41:05.606555 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tnrxd" event={"ID":"cca60bdc-e638-44e1-a091-9958e5de0c23","Type":"ContainerStarted","Data":"f3ef1021c4a84f7b5942023dbab023d2c9bfabc26f8d6aa6487855db4301e6ff"} Jul 08 17:41:08.447710 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:41:08.447676 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:22.491787 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:41:22.491750 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:34.552042 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:41:34.552004 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:35.194833 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:41:35.194794 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:46.485144 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:41:46.485050 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:46.544138 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:41:46.544105 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:53.030395 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:41:53.030353 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:41:53.089422 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:41:53.089382 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:00.546903 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:00.546867 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:00.644378 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:00.644344 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:03.813296 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:03.813263 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:03.915837 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:03.915800 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:03.992590 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:03.992550 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:30.755110 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:30.755052 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:30.811985 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:30.811939 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:36.970131 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:36.970089 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:39.849435 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:39.849386 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:39.950427 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:39.950394 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:42:40.443846 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.443809 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx"] Jul 08 17:42:40.444094 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.444082 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="278faa32-e840-4525-a31d-cf894fdc4b2d" containerName="registry" Jul 08 17:42:40.444137 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.444095 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="278faa32-e840-4525-a31d-cf894fdc4b2d" containerName="registry" Jul 08 17:42:40.444169 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.444149 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="278faa32-e840-4525-a31d-cf894fdc4b2d" containerName="registry" Jul 08 17:42:40.447412 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.447388 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx" Jul 08 17:42:40.449807 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.449783 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 08 17:42:40.450634 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.450616 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 08 17:42:40.450719 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.450619 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jr4fm\"" Jul 08 17:42:40.456116 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.456091 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx"] Jul 08 17:42:40.550531 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.550492 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2560cd8d-5cca-4d64-b91b-7b9f4171e3df-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx\" (UID: \"2560cd8d-5cca-4d64-b91b-7b9f4171e3df\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx" Jul 08 17:42:40.550732 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.550552 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2560cd8d-5cca-4d64-b91b-7b9f4171e3df-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx\" (UID: \"2560cd8d-5cca-4d64-b91b-7b9f4171e3df\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx" Jul 08 17:42:40.550732 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.550574 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxg6p\" (UniqueName: \"kubernetes.io/projected/2560cd8d-5cca-4d64-b91b-7b9f4171e3df-kube-api-access-dxg6p\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx\" (UID: \"2560cd8d-5cca-4d64-b91b-7b9f4171e3df\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx" Jul 08 17:42:40.651882 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.651833 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2560cd8d-5cca-4d64-b91b-7b9f4171e3df-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx\" (UID: \"2560cd8d-5cca-4d64-b91b-7b9f4171e3df\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx" Jul 08 17:42:40.651882 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.651883 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxg6p\" (UniqueName: \"kubernetes.io/projected/2560cd8d-5cca-4d64-b91b-7b9f4171e3df-kube-api-access-dxg6p\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx\" (UID: \"2560cd8d-5cca-4d64-b91b-7b9f4171e3df\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx" Jul 08 17:42:40.652145 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.651924 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2560cd8d-5cca-4d64-b91b-7b9f4171e3df-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx\" (UID: \"2560cd8d-5cca-4d64-b91b-7b9f4171e3df\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx" Jul 08 17:42:40.652346 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.652323 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2560cd8d-5cca-4d64-b91b-7b9f4171e3df-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx\" (UID: \"2560cd8d-5cca-4d64-b91b-7b9f4171e3df\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx" Jul 08 17:42:40.652428 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.652345 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2560cd8d-5cca-4d64-b91b-7b9f4171e3df-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx\" (UID: \"2560cd8d-5cca-4d64-b91b-7b9f4171e3df\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx" Jul 08 17:42:40.662083 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.662046 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxg6p\" (UniqueName: \"kubernetes.io/projected/2560cd8d-5cca-4d64-b91b-7b9f4171e3df-kube-api-access-dxg6p\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx\" (UID: \"2560cd8d-5cca-4d64-b91b-7b9f4171e3df\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx" Jul 08 17:42:40.757077 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.756999 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx" Jul 08 17:42:40.885979 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.885880 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx"] Jul 08 17:42:40.888724 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:42:40.888692 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2560cd8d_5cca_4d64_b91b_7b9f4171e3df.slice/crio-54640304337b5406fe33ae6573240c35332975af685a3d44fedadc22ba5bb03c WatchSource:0}: Error finding container 54640304337b5406fe33ae6573240c35332975af685a3d44fedadc22ba5bb03c: Status 404 returned error can't find the container with id 54640304337b5406fe33ae6573240c35332975af685a3d44fedadc22ba5bb03c Jul 08 17:42:40.955275 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:40.955225 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx" event={"ID":"2560cd8d-5cca-4d64-b91b-7b9f4171e3df","Type":"ContainerStarted","Data":"54640304337b5406fe33ae6573240c35332975af685a3d44fedadc22ba5bb03c"} Jul 08 17:42:48.987146 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:48.987103 2584 generic.go:356] "Generic (PLEG): container finished" podID="2560cd8d-5cca-4d64-b91b-7b9f4171e3df" containerID="e6d1d8b60b7e92c59de3b2e915855f91d9031b8024fcb6d926f97ed0e31eb90b" exitCode=0 Jul 08 17:42:48.987584 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:48.987183 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx" event={"ID":"2560cd8d-5cca-4d64-b91b-7b9f4171e3df","Type":"ContainerDied","Data":"e6d1d8b60b7e92c59de3b2e915855f91d9031b8024fcb6d926f97ed0e31eb90b"} Jul 08 17:42:50.996098 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:50.996060 2584 generic.go:356] "Generic (PLEG): container finished" podID="2560cd8d-5cca-4d64-b91b-7b9f4171e3df" containerID="e329a210491837a6c27ee7b74ac3c8ea9a9bd05657a6dc230eef03879d73af7d" exitCode=0 Jul 08 17:42:50.996509 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:42:50.996147 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx" event={"ID":"2560cd8d-5cca-4d64-b91b-7b9f4171e3df","Type":"ContainerDied","Data":"e329a210491837a6c27ee7b74ac3c8ea9a9bd05657a6dc230eef03879d73af7d"} Jul 08 17:43:02.037236 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:02.037191 2584 generic.go:356] "Generic (PLEG): container finished" podID="2560cd8d-5cca-4d64-b91b-7b9f4171e3df" containerID="965c6d9eba69a544662bca8fc80200231ee505256c9677e5f5604bb924379ab8" exitCode=0 Jul 08 17:43:02.037658 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:02.037259 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx" event={"ID":"2560cd8d-5cca-4d64-b91b-7b9f4171e3df","Type":"ContainerDied","Data":"965c6d9eba69a544662bca8fc80200231ee505256c9677e5f5604bb924379ab8"} Jul 08 17:43:03.216845 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:03.216812 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx" Jul 08 17:43:03.342386 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:03.342295 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxg6p\" (UniqueName: \"kubernetes.io/projected/2560cd8d-5cca-4d64-b91b-7b9f4171e3df-kube-api-access-dxg6p\") pod \"2560cd8d-5cca-4d64-b91b-7b9f4171e3df\" (UID: \"2560cd8d-5cca-4d64-b91b-7b9f4171e3df\") " Jul 08 17:43:03.342386 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:03.342368 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2560cd8d-5cca-4d64-b91b-7b9f4171e3df-bundle\") pod \"2560cd8d-5cca-4d64-b91b-7b9f4171e3df\" (UID: \"2560cd8d-5cca-4d64-b91b-7b9f4171e3df\") " Jul 08 17:43:03.342630 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:03.342394 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2560cd8d-5cca-4d64-b91b-7b9f4171e3df-util\") pod \"2560cd8d-5cca-4d64-b91b-7b9f4171e3df\" (UID: \"2560cd8d-5cca-4d64-b91b-7b9f4171e3df\") " Jul 08 17:43:03.343123 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:03.343092 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2560cd8d-5cca-4d64-b91b-7b9f4171e3df-bundle" (OuterVolumeSpecName: "bundle") pod "2560cd8d-5cca-4d64-b91b-7b9f4171e3df" (UID: "2560cd8d-5cca-4d64-b91b-7b9f4171e3df"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:43:03.344545 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:03.344522 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2560cd8d-5cca-4d64-b91b-7b9f4171e3df-kube-api-access-dxg6p" (OuterVolumeSpecName: "kube-api-access-dxg6p") pod "2560cd8d-5cca-4d64-b91b-7b9f4171e3df" (UID: "2560cd8d-5cca-4d64-b91b-7b9f4171e3df"). InnerVolumeSpecName "kube-api-access-dxg6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:43:03.347028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:03.347004 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2560cd8d-5cca-4d64-b91b-7b9f4171e3df-util" (OuterVolumeSpecName: "util") pod "2560cd8d-5cca-4d64-b91b-7b9f4171e3df" (UID: "2560cd8d-5cca-4d64-b91b-7b9f4171e3df"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:43:03.443127 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:03.443086 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxg6p\" (UniqueName: \"kubernetes.io/projected/2560cd8d-5cca-4d64-b91b-7b9f4171e3df-kube-api-access-dxg6p\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:43:03.443127 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:03.443121 2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2560cd8d-5cca-4d64-b91b-7b9f4171e3df-bundle\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:43:03.443127 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:03.443130 2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2560cd8d-5cca-4d64-b91b-7b9f4171e3df-util\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:43:04.045597 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:04.045571 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx" Jul 08 17:43:04.045764 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:04.045573 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540g46vx" event={"ID":"2560cd8d-5cca-4d64-b91b-7b9f4171e3df","Type":"ContainerDied","Data":"54640304337b5406fe33ae6573240c35332975af685a3d44fedadc22ba5bb03c"} Jul 08 17:43:04.045764 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:04.045699 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54640304337b5406fe33ae6573240c35332975af685a3d44fedadc22ba5bb03c" Jul 08 17:43:06.319145 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:06.319090 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:07.681004 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:07.680877 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:08.244543 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.244511 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9d8zh"] Jul 08 17:43:08.244782 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.244771 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2560cd8d-5cca-4d64-b91b-7b9f4171e3df" containerName="util" Jul 08 17:43:08.244824 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.244783 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2560cd8d-5cca-4d64-b91b-7b9f4171e3df" containerName="util" Jul 08 17:43:08.244824 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.244793 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2560cd8d-5cca-4d64-b91b-7b9f4171e3df" containerName="extract" Jul 08 17:43:08.244824 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.244798 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2560cd8d-5cca-4d64-b91b-7b9f4171e3df" containerName="extract" Jul 08 17:43:08.244824 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.244812 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2560cd8d-5cca-4d64-b91b-7b9f4171e3df" containerName="pull" Jul 08 17:43:08.244824 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.244817 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2560cd8d-5cca-4d64-b91b-7b9f4171e3df" containerName="pull" Jul 08 17:43:08.245013 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.244860 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2560cd8d-5cca-4d64-b91b-7b9f4171e3df" containerName="extract" Jul 08 17:43:08.247609 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.247593 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9d8zh" Jul 08 17:43:08.250381 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.250359 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 08 17:43:08.250381 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.250375 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-kx765\"" Jul 08 17:43:08.250526 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.250361 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 08 17:43:08.260163 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.260142 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9d8zh"] Jul 08 17:43:08.387220 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.387180 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnnjd\" (UniqueName: \"kubernetes.io/projected/d64cc0f2-75c7-41c1-bcc7-109870e6114c-kube-api-access-wnnjd\") pod \"cert-manager-operator-controller-manager-55c444bf86-9d8zh\" (UID: \"d64cc0f2-75c7-41c1-bcc7-109870e6114c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9d8zh" Jul 08 17:43:08.387409 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.387263 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d64cc0f2-75c7-41c1-bcc7-109870e6114c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9d8zh\" (UID: \"d64cc0f2-75c7-41c1-bcc7-109870e6114c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9d8zh" Jul 08 17:43:08.488213 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.488177 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d64cc0f2-75c7-41c1-bcc7-109870e6114c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9d8zh\" (UID: \"d64cc0f2-75c7-41c1-bcc7-109870e6114c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9d8zh" Jul 08 17:43:08.488408 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.488260 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnnjd\" (UniqueName: \"kubernetes.io/projected/d64cc0f2-75c7-41c1-bcc7-109870e6114c-kube-api-access-wnnjd\") pod \"cert-manager-operator-controller-manager-55c444bf86-9d8zh\" (UID: \"d64cc0f2-75c7-41c1-bcc7-109870e6114c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9d8zh" Jul 08 17:43:08.488593 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.488573 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d64cc0f2-75c7-41c1-bcc7-109870e6114c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9d8zh\" (UID: \"d64cc0f2-75c7-41c1-bcc7-109870e6114c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9d8zh" Jul 08 17:43:08.499154 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.499100 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnnjd\" (UniqueName: \"kubernetes.io/projected/d64cc0f2-75c7-41c1-bcc7-109870e6114c-kube-api-access-wnnjd\") pod \"cert-manager-operator-controller-manager-55c444bf86-9d8zh\" (UID: \"d64cc0f2-75c7-41c1-bcc7-109870e6114c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9d8zh" Jul 08 17:43:08.556687 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.556655 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9d8zh" Jul 08 17:43:08.692092 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:08.692064 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9d8zh"] Jul 08 17:43:08.694915 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:43:08.694886 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd64cc0f2_75c7_41c1_bcc7_109870e6114c.slice/crio-af022c4754de596f4c5645212569bb0dfc49f85cd0dcfd3b81934b3239c4cba0 WatchSource:0}: Error finding container af022c4754de596f4c5645212569bb0dfc49f85cd0dcfd3b81934b3239c4cba0: Status 404 returned error can't find the container with id af022c4754de596f4c5645212569bb0dfc49f85cd0dcfd3b81934b3239c4cba0 Jul 08 17:43:09.065034 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:09.064998 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9d8zh" event={"ID":"d64cc0f2-75c7-41c1-bcc7-109870e6114c","Type":"ContainerStarted","Data":"af022c4754de596f4c5645212569bb0dfc49f85cd0dcfd3b81934b3239c4cba0"} Jul 08 17:43:09.932420 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:09.932379 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:09.989208 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:09.989170 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:13.862746 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:13.862708 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:13.964750 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:13.964712 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:14.087149 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:14.087107 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9d8zh" event={"ID":"d64cc0f2-75c7-41c1-bcc7-109870e6114c","Type":"ContainerStarted","Data":"f6cc4be023722b4d09f0617f7eb0c934b619c0416eeb749a6d44617ee9cdd016"} Jul 08 17:43:14.109189 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:14.109139 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9d8zh" podStartSLOduration=1.6132611639999999 podStartE2EDuration="6.109122842s" totalImagesPullingTime="4.495861678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:43:14.107752558 +0000 UTC m=+269.791849835" watchObservedRunningTime="2026-07-08 17:43:14.109122842 +0000 UTC m=+269.793220120" Jul 08 17:43:14.474513 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:14.474469 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:14.571464 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:14.571423 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:15.320826 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:15.320787 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:15.428019 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:15.427978 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:15.944302 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:15.944266 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29"] Jul 08 17:43:15.947730 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:15.947712 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29" Jul 08 17:43:15.950343 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:15.950321 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jr4fm\"" Jul 08 17:43:15.950463 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:15.950328 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 08 17:43:15.950463 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:15.950332 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 08 17:43:15.955618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:15.955598 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29"] Jul 08 17:43:16.048197 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.048162 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a2bd493-a047-4d27-ac39-7b9a9c19cf6c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29\" (UID: \"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29" Jul 08 17:43:16.048420 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.048211 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a2bd493-a047-4d27-ac39-7b9a9c19cf6c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29\" (UID: \"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29" Jul 08 17:43:16.048420 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.048274 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhfsm\" (UniqueName: \"kubernetes.io/projected/7a2bd493-a047-4d27-ac39-7b9a9c19cf6c-kube-api-access-vhfsm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29\" (UID: \"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29" Jul 08 17:43:16.149757 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.149716 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a2bd493-a047-4d27-ac39-7b9a9c19cf6c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29\" (UID: \"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29" Jul 08 17:43:16.149933 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.149775 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a2bd493-a047-4d27-ac39-7b9a9c19cf6c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29\" (UID: \"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29" Jul 08 17:43:16.149933 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.149820 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhfsm\" (UniqueName: \"kubernetes.io/projected/7a2bd493-a047-4d27-ac39-7b9a9c19cf6c-kube-api-access-vhfsm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29\" (UID: \"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29" Jul 08 17:43:16.150197 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.150173 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a2bd493-a047-4d27-ac39-7b9a9c19cf6c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29\" (UID: \"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29" Jul 08 17:43:16.150265 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.150183 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a2bd493-a047-4d27-ac39-7b9a9c19cf6c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29\" (UID: \"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29" Jul 08 17:43:16.159694 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.159666 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhfsm\" (UniqueName: \"kubernetes.io/projected/7a2bd493-a047-4d27-ac39-7b9a9c19cf6c-kube-api-access-vhfsm\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29\" (UID: \"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29" Jul 08 17:43:16.257379 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.257291 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29" Jul 08 17:43:16.389428 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.389403 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29"] Jul 08 17:43:16.391762 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:43:16.391730 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2bd493_a047_4d27_ac39_7b9a9c19cf6c.slice/crio-68ea700a28353d126e48ffdbbf044e3ba38c0e17c509a47b36bc333f8ab48b49 WatchSource:0}: Error finding container 68ea700a28353d126e48ffdbbf044e3ba38c0e17c509a47b36bc333f8ab48b49: Status 404 returned error can't find the container with id 68ea700a28353d126e48ffdbbf044e3ba38c0e17c509a47b36bc333f8ab48b49 Jul 08 17:43:16.401843 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.401811 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:16.451228 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.451197 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-n59gj"] Jul 08 17:43:16.454416 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.454400 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-n59gj" Jul 08 17:43:16.456761 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.456732 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 08 17:43:16.457131 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.457058 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 08 17:43:16.457260 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.457187 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-4mgb6\"" Jul 08 17:43:16.465274 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.465229 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-n59gj"] Jul 08 17:43:16.519067 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.518980 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:16.553643 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.553615 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/421dde25-7d52-4abc-8a47-9a72197f9ccb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-n59gj\" (UID: \"421dde25-7d52-4abc-8a47-9a72197f9ccb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n59gj" Jul 08 17:43:16.553788 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.553665 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m247n\" (UniqueName: \"kubernetes.io/projected/421dde25-7d52-4abc-8a47-9a72197f9ccb-kube-api-access-m247n\") pod \"cert-manager-webhook-6486ffd997-n59gj\" (UID: \"421dde25-7d52-4abc-8a47-9a72197f9ccb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n59gj" Jul 08 17:43:16.655036 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.654998 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/421dde25-7d52-4abc-8a47-9a72197f9ccb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-n59gj\" (UID: \"421dde25-7d52-4abc-8a47-9a72197f9ccb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n59gj" Jul 08 17:43:16.655036 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.655039 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m247n\" (UniqueName: \"kubernetes.io/projected/421dde25-7d52-4abc-8a47-9a72197f9ccb-kube-api-access-m247n\") pod \"cert-manager-webhook-6486ffd997-n59gj\" (UID: \"421dde25-7d52-4abc-8a47-9a72197f9ccb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n59gj" Jul 08 17:43:16.666977 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.666945 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/421dde25-7d52-4abc-8a47-9a72197f9ccb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-n59gj\" (UID: \"421dde25-7d52-4abc-8a47-9a72197f9ccb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n59gj" Jul 08 17:43:16.667114 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.667089 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m247n\" (UniqueName: \"kubernetes.io/projected/421dde25-7d52-4abc-8a47-9a72197f9ccb-kube-api-access-m247n\") pod \"cert-manager-webhook-6486ffd997-n59gj\" (UID: \"421dde25-7d52-4abc-8a47-9a72197f9ccb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-n59gj" Jul 08 17:43:16.772797 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.772721 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-n59gj" Jul 08 17:43:16.900818 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:16.900790 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-n59gj"] Jul 08 17:43:16.903068 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:43:16.903037 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod421dde25_7d52_4abc_8a47_9a72197f9ccb.slice/crio-ff901b28d0544879fa893f921c97f1f6edf001c869ced7d05c6a75a66ed68ff9 WatchSource:0}: Error finding container ff901b28d0544879fa893f921c97f1f6edf001c869ced7d05c6a75a66ed68ff9: Status 404 returned error can't find the container with id ff901b28d0544879fa893f921c97f1f6edf001c869ced7d05c6a75a66ed68ff9 Jul 08 17:43:17.100205 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:17.100116 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-n59gj" event={"ID":"421dde25-7d52-4abc-8a47-9a72197f9ccb","Type":"ContainerStarted","Data":"ff901b28d0544879fa893f921c97f1f6edf001c869ced7d05c6a75a66ed68ff9"} Jul 08 17:43:17.101506 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:17.101480 2584 generic.go:356] "Generic (PLEG): container finished" podID="7a2bd493-a047-4d27-ac39-7b9a9c19cf6c" containerID="1047a47bc9aaa256ab49d148fd061f79b7adf894b820d68d8c430288c698362d" exitCode=0 Jul 08 17:43:17.101595 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:17.101522 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29" event={"ID":"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c","Type":"ContainerDied","Data":"1047a47bc9aaa256ab49d148fd061f79b7adf894b820d68d8c430288c698362d"} Jul 08 17:43:17.101595 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:17.101558 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29" event={"ID":"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c","Type":"ContainerStarted","Data":"68ea700a28353d126e48ffdbbf044e3ba38c0e17c509a47b36bc333f8ab48b49"} Jul 08 17:43:20.118056 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:20.118023 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-n59gj" event={"ID":"421dde25-7d52-4abc-8a47-9a72197f9ccb","Type":"ContainerStarted","Data":"3f5cf50f5e9f2e52b09bb097c6fdc9718fff7fe27742d098052a1de853d4d01c"} Jul 08 17:43:20.118505 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:20.118098 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-n59gj" Jul 08 17:43:20.119738 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:20.119713 2584 generic.go:356] "Generic (PLEG): container finished" podID="7a2bd493-a047-4d27-ac39-7b9a9c19cf6c" containerID="7ff51e3daac12708c8b21fb44f53fd27e39d9d2bfa9001435ba4598c37afae2b" exitCode=0 Jul 08 17:43:20.119803 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:20.119761 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29" event={"ID":"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c","Type":"ContainerDied","Data":"7ff51e3daac12708c8b21fb44f53fd27e39d9d2bfa9001435ba4598c37afae2b"} Jul 08 17:43:20.141887 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:20.141836 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-n59gj" podStartSLOduration=1.170157731 podStartE2EDuration="4.14181994s" totalImagesPullingTime="2.971662209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:43:20.141027434 +0000 UTC m=+275.825124712" watchObservedRunningTime="2026-07-08 17:43:20.14181994 +0000 UTC m=+275.825917217" Jul 08 17:43:21.125432 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:21.125393 2584 generic.go:356] "Generic (PLEG): container finished" podID="7a2bd493-a047-4d27-ac39-7b9a9c19cf6c" containerID="0a0b437f50208c17cd542bde35dafd23b94ad4e5bab320f604a953c8f132af46" exitCode=0 Jul 08 17:43:21.125914 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:21.125485 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29" event={"ID":"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c","Type":"ContainerDied","Data":"0a0b437f50208c17cd542bde35dafd23b94ad4e5bab320f604a953c8f132af46"} Jul 08 17:43:21.994700 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:21.994670 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:22.094036 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:22.094001 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:22.257502 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:22.257441 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29" Jul 08 17:43:22.301284 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:22.301220 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a2bd493-a047-4d27-ac39-7b9a9c19cf6c-util\") pod \"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c\" (UID: \"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c\") " Jul 08 17:43:22.301469 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:22.301303 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhfsm\" (UniqueName: \"kubernetes.io/projected/7a2bd493-a047-4d27-ac39-7b9a9c19cf6c-kube-api-access-vhfsm\") pod \"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c\" (UID: \"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c\") " Jul 08 17:43:22.301469 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:22.301332 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a2bd493-a047-4d27-ac39-7b9a9c19cf6c-bundle\") pod \"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c\" (UID: \"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c\") " Jul 08 17:43:22.301756 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:22.301729 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2bd493-a047-4d27-ac39-7b9a9c19cf6c-bundle" (OuterVolumeSpecName: "bundle") pod "7a2bd493-a047-4d27-ac39-7b9a9c19cf6c" (UID: "7a2bd493-a047-4d27-ac39-7b9a9c19cf6c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:43:22.303462 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:22.303432 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2bd493-a047-4d27-ac39-7b9a9c19cf6c-kube-api-access-vhfsm" (OuterVolumeSpecName: "kube-api-access-vhfsm") pod "7a2bd493-a047-4d27-ac39-7b9a9c19cf6c" (UID: "7a2bd493-a047-4d27-ac39-7b9a9c19cf6c"). InnerVolumeSpecName "kube-api-access-vhfsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:43:22.306276 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:22.306226 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a2bd493-a047-4d27-ac39-7b9a9c19cf6c-util" (OuterVolumeSpecName: "util") pod "7a2bd493-a047-4d27-ac39-7b9a9c19cf6c" (UID: "7a2bd493-a047-4d27-ac39-7b9a9c19cf6c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:43:22.402359 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:22.402324 2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7a2bd493-a047-4d27-ac39-7b9a9c19cf6c-util\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:43:22.402359 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:22.402351 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhfsm\" (UniqueName: \"kubernetes.io/projected/7a2bd493-a047-4d27-ac39-7b9a9c19cf6c-kube-api-access-vhfsm\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:43:22.402359 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:22.402361 2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7a2bd493-a047-4d27-ac39-7b9a9c19cf6c-bundle\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:43:23.134797 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:23.134760 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29" event={"ID":"7a2bd493-a047-4d27-ac39-7b9a9c19cf6c","Type":"ContainerDied","Data":"68ea700a28353d126e48ffdbbf044e3ba38c0e17c509a47b36bc333f8ab48b49"} Jul 08 17:43:23.134797 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:23.134801 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxpv29" Jul 08 17:43:23.135007 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:23.134814 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68ea700a28353d126e48ffdbbf044e3ba38c0e17c509a47b36bc333f8ab48b49" Jul 08 17:43:23.508776 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:23.508742 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:26.127654 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:26.127617 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-n59gj" Jul 08 17:43:28.005168 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:28.005130 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:28.104330 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:28.104292 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:29.407695 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.407661 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8xr9f"] Jul 08 17:43:29.408094 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.407936 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a2bd493-a047-4d27-ac39-7b9a9c19cf6c" containerName="extract" Jul 08 17:43:29.408094 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.407947 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a2bd493-a047-4d27-ac39-7b9a9c19cf6c" containerName="extract" Jul 08 17:43:29.408094 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.407960 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a2bd493-a047-4d27-ac39-7b9a9c19cf6c" containerName="pull" Jul 08 17:43:29.408094 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.407965 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a2bd493-a047-4d27-ac39-7b9a9c19cf6c" containerName="pull" Jul 08 17:43:29.408094 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.407972 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a2bd493-a047-4d27-ac39-7b9a9c19cf6c" containerName="util" Jul 08 17:43:29.408094 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.407978 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a2bd493-a047-4d27-ac39-7b9a9c19cf6c" containerName="util" Jul 08 17:43:29.408094 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.408036 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a2bd493-a047-4d27-ac39-7b9a9c19cf6c" containerName="extract" Jul 08 17:43:29.412272 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.412229 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8xr9f" Jul 08 17:43:29.415166 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.415145 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-xtpqg\"" Jul 08 17:43:29.415272 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.415151 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 08 17:43:29.415762 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.415745 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 08 17:43:29.421130 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.421110 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8xr9f"] Jul 08 17:43:29.454962 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.454927 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89020466-4f77-4a18-8f7c-a6570a2640b8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8xr9f\" (UID: \"89020466-4f77-4a18-8f7c-a6570a2640b8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8xr9f" Jul 08 17:43:29.455121 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.455000 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfm4c\" (UniqueName: \"kubernetes.io/projected/89020466-4f77-4a18-8f7c-a6570a2640b8-kube-api-access-hfm4c\") pod \"openshift-lws-operator-fd8ccff4c-8xr9f\" (UID: \"89020466-4f77-4a18-8f7c-a6570a2640b8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8xr9f" Jul 08 17:43:29.555486 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.555444 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89020466-4f77-4a18-8f7c-a6570a2640b8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8xr9f\" (UID: \"89020466-4f77-4a18-8f7c-a6570a2640b8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8xr9f" Jul 08 17:43:29.555679 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.555536 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfm4c\" (UniqueName: \"kubernetes.io/projected/89020466-4f77-4a18-8f7c-a6570a2640b8-kube-api-access-hfm4c\") pod \"openshift-lws-operator-fd8ccff4c-8xr9f\" (UID: \"89020466-4f77-4a18-8f7c-a6570a2640b8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8xr9f" Jul 08 17:43:29.555842 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.555818 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/89020466-4f77-4a18-8f7c-a6570a2640b8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8xr9f\" (UID: \"89020466-4f77-4a18-8f7c-a6570a2640b8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8xr9f" Jul 08 17:43:29.565614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.565583 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfm4c\" (UniqueName: \"kubernetes.io/projected/89020466-4f77-4a18-8f7c-a6570a2640b8-kube-api-access-hfm4c\") pod \"openshift-lws-operator-fd8ccff4c-8xr9f\" (UID: \"89020466-4f77-4a18-8f7c-a6570a2640b8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8xr9f" Jul 08 17:43:29.721094 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.721000 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8xr9f" Jul 08 17:43:29.856992 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:29.856963 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8xr9f"] Jul 08 17:43:29.859491 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:43:29.859451 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89020466_4f77_4a18_8f7c_a6570a2640b8.slice/crio-b4d6d6291b57fcc17f6cec1cf9b8c4be888fedb52bcc2305c0de334a9ab1cc37 WatchSource:0}: Error finding container b4d6d6291b57fcc17f6cec1cf9b8c4be888fedb52bcc2305c0de334a9ab1cc37: Status 404 returned error can't find the container with id b4d6d6291b57fcc17f6cec1cf9b8c4be888fedb52bcc2305c0de334a9ab1cc37 Jul 08 17:43:30.165369 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:30.165334 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8xr9f" event={"ID":"89020466-4f77-4a18-8f7c-a6570a2640b8","Type":"ContainerStarted","Data":"b4d6d6291b57fcc17f6cec1cf9b8c4be888fedb52bcc2305c0de334a9ab1cc37"} Jul 08 17:43:32.176191 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:32.176099 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8xr9f" event={"ID":"89020466-4f77-4a18-8f7c-a6570a2640b8","Type":"ContainerStarted","Data":"e7f6dfec58569ffb855e83e5f55240bd33bf3ebf458f442dccee9becb9d15a94"} Jul 08 17:43:32.196878 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:32.196805 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8xr9f" podStartSLOduration=1.570417676 podStartE2EDuration="3.196788414s" totalImagesPullingTime="1.626370738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:43:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:43:32.194863476 +0000 UTC m=+287.878960753" watchObservedRunningTime="2026-07-08 17:43:32.196788414 +0000 UTC m=+287.880885692" Jul 08 17:43:32.909093 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:32.909058 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:33.012918 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:33.012878 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:34.708402 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:34.708365 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:34.831671 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:34.831636 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929"] Jul 08 17:43:34.875081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:34.875047 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929"] Jul 08 17:43:34.875273 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:34.875156 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929" Jul 08 17:43:34.877931 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:34.877900 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 08 17:43:34.878143 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:34.878117 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 08 17:43:34.878143 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:34.878135 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jr4fm\"" Jul 08 17:43:34.908905 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:34.908870 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51d3d69f-ff17-432a-a0ce-0c2a87601858-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929\" (UID: \"51d3d69f-ff17-432a-a0ce-0c2a87601858\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929" Jul 08 17:43:34.908905 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:34.908915 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51d3d69f-ff17-432a-a0ce-0c2a87601858-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929\" (UID: \"51d3d69f-ff17-432a-a0ce-0c2a87601858\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929" Jul 08 17:43:34.909158 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:34.909032 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sdzr\" (UniqueName: \"kubernetes.io/projected/51d3d69f-ff17-432a-a0ce-0c2a87601858-kube-api-access-2sdzr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929\" (UID: \"51d3d69f-ff17-432a-a0ce-0c2a87601858\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929" Jul 08 17:43:35.010109 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:35.010004 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sdzr\" (UniqueName: \"kubernetes.io/projected/51d3d69f-ff17-432a-a0ce-0c2a87601858-kube-api-access-2sdzr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929\" (UID: \"51d3d69f-ff17-432a-a0ce-0c2a87601858\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929" Jul 08 17:43:35.010109 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:35.010066 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51d3d69f-ff17-432a-a0ce-0c2a87601858-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929\" (UID: \"51d3d69f-ff17-432a-a0ce-0c2a87601858\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929" Jul 08 17:43:35.010109 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:35.010096 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51d3d69f-ff17-432a-a0ce-0c2a87601858-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929\" (UID: \"51d3d69f-ff17-432a-a0ce-0c2a87601858\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929" Jul 08 17:43:35.010521 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:35.010491 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51d3d69f-ff17-432a-a0ce-0c2a87601858-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929\" (UID: \"51d3d69f-ff17-432a-a0ce-0c2a87601858\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929" Jul 08 17:43:35.010580 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:35.010538 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51d3d69f-ff17-432a-a0ce-0c2a87601858-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929\" (UID: \"51d3d69f-ff17-432a-a0ce-0c2a87601858\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929" Jul 08 17:43:35.020114 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:35.020081 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sdzr\" (UniqueName: \"kubernetes.io/projected/51d3d69f-ff17-432a-a0ce-0c2a87601858-kube-api-access-2sdzr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929\" (UID: \"51d3d69f-ff17-432a-a0ce-0c2a87601858\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929" Jul 08 17:43:35.185847 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:35.185803 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929" Jul 08 17:43:35.343113 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:35.343087 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929"] Jul 08 17:43:35.345112 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:43:35.345072 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d3d69f_ff17_432a_a0ce_0c2a87601858.slice/crio-68d2ac344ad98e011ef41f71ac16f965f833ea4ac51958f93f98d7106b2ce28c WatchSource:0}: Error finding container 68d2ac344ad98e011ef41f71ac16f965f833ea4ac51958f93f98d7106b2ce28c: Status 404 returned error can't find the container with id 68d2ac344ad98e011ef41f71ac16f965f833ea4ac51958f93f98d7106b2ce28c Jul 08 17:43:36.193768 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:36.193731 2584 generic.go:356] "Generic (PLEG): container finished" podID="51d3d69f-ff17-432a-a0ce-0c2a87601858" containerID="8f718162f35d998da56745fce83cb557f832a5ab0df33eabc10b5ce19fd33e85" exitCode=0 Jul 08 17:43:36.194216 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:36.193778 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929" event={"ID":"51d3d69f-ff17-432a-a0ce-0c2a87601858","Type":"ContainerDied","Data":"8f718162f35d998da56745fce83cb557f832a5ab0df33eabc10b5ce19fd33e85"} Jul 08 17:43:36.194216 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:36.193849 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929" event={"ID":"51d3d69f-ff17-432a-a0ce-0c2a87601858","Type":"ContainerStarted","Data":"68d2ac344ad98e011ef41f71ac16f965f833ea4ac51958f93f98d7106b2ce28c"} Jul 08 17:43:37.200068 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:37.200027 2584 generic.go:356] "Generic (PLEG): container finished" podID="51d3d69f-ff17-432a-a0ce-0c2a87601858" containerID="ed682e68dedf3adf1ec29fe11e45e59d307d0a97c13b869722996ef450115d58" exitCode=0 Jul 08 17:43:37.200438 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:37.200079 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929" event={"ID":"51d3d69f-ff17-432a-a0ce-0c2a87601858","Type":"ContainerDied","Data":"ed682e68dedf3adf1ec29fe11e45e59d307d0a97c13b869722996ef450115d58"} Jul 08 17:43:38.207547 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:38.207509 2584 generic.go:356] "Generic (PLEG): container finished" podID="51d3d69f-ff17-432a-a0ce-0c2a87601858" containerID="90d61bedd7b70a0d37a561d3a04a268aecff3307eaa4e989da720000dc690aa9" exitCode=0 Jul 08 17:43:38.207914 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:38.207588 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929" event={"ID":"51d3d69f-ff17-432a-a0ce-0c2a87601858","Type":"ContainerDied","Data":"90d61bedd7b70a0d37a561d3a04a268aecff3307eaa4e989da720000dc690aa9"} Jul 08 17:43:39.342291 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:39.342267 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929" Jul 08 17:43:39.448114 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:39.448084 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51d3d69f-ff17-432a-a0ce-0c2a87601858-util\") pod \"51d3d69f-ff17-432a-a0ce-0c2a87601858\" (UID: \"51d3d69f-ff17-432a-a0ce-0c2a87601858\") " Jul 08 17:43:39.448308 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:39.448130 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sdzr\" (UniqueName: \"kubernetes.io/projected/51d3d69f-ff17-432a-a0ce-0c2a87601858-kube-api-access-2sdzr\") pod \"51d3d69f-ff17-432a-a0ce-0c2a87601858\" (UID: \"51d3d69f-ff17-432a-a0ce-0c2a87601858\") " Jul 08 17:43:39.448308 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:39.448200 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51d3d69f-ff17-432a-a0ce-0c2a87601858-bundle\") pod \"51d3d69f-ff17-432a-a0ce-0c2a87601858\" (UID: \"51d3d69f-ff17-432a-a0ce-0c2a87601858\") " Jul 08 17:43:39.448851 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:39.448821 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d3d69f-ff17-432a-a0ce-0c2a87601858-bundle" (OuterVolumeSpecName: "bundle") pod "51d3d69f-ff17-432a-a0ce-0c2a87601858" (UID: "51d3d69f-ff17-432a-a0ce-0c2a87601858"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:43:39.450169 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:39.450144 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d3d69f-ff17-432a-a0ce-0c2a87601858-kube-api-access-2sdzr" (OuterVolumeSpecName: "kube-api-access-2sdzr") pod "51d3d69f-ff17-432a-a0ce-0c2a87601858" (UID: "51d3d69f-ff17-432a-a0ce-0c2a87601858"). InnerVolumeSpecName "kube-api-access-2sdzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:43:39.453446 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:39.453413 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d3d69f-ff17-432a-a0ce-0c2a87601858-util" (OuterVolumeSpecName: "util") pod "51d3d69f-ff17-432a-a0ce-0c2a87601858" (UID: "51d3d69f-ff17-432a-a0ce-0c2a87601858"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:43:39.549302 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:39.549207 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sdzr\" (UniqueName: \"kubernetes.io/projected/51d3d69f-ff17-432a-a0ce-0c2a87601858-kube-api-access-2sdzr\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:43:39.549302 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:39.549273 2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/51d3d69f-ff17-432a-a0ce-0c2a87601858-bundle\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:43:39.549302 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:39.549283 2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/51d3d69f-ff17-432a-a0ce-0c2a87601858-util\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:43:40.218202 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:40.218169 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929" event={"ID":"51d3d69f-ff17-432a-a0ce-0c2a87601858","Type":"ContainerDied","Data":"68d2ac344ad98e011ef41f71ac16f965f833ea4ac51958f93f98d7106b2ce28c"} Jul 08 17:43:40.218388 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:40.218216 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68d2ac344ad98e011ef41f71ac16f965f833ea4ac51958f93f98d7106b2ce28c" Jul 08 17:43:40.218388 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:40.218183 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afl9929" Jul 08 17:43:44.689486 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:44.689435 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:44.759683 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:44.759655 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/1.log" Jul 08 17:43:44.765092 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:44.765058 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/1.log" Jul 08 17:43:44.771769 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:44.771741 2584 kubelet.go:1648] "Image garbage collection succeeded" Jul 08 17:43:50.968076 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:50.968037 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:52.531792 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.531755 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:52.571628 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.571592 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt"] Jul 08 17:43:52.571886 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.571874 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51d3d69f-ff17-432a-a0ce-0c2a87601858" containerName="extract" Jul 08 17:43:52.571931 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.571887 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="51d3d69f-ff17-432a-a0ce-0c2a87601858" containerName="extract" Jul 08 17:43:52.571931 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.571901 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51d3d69f-ff17-432a-a0ce-0c2a87601858" containerName="util" Jul 08 17:43:52.571931 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.571906 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="51d3d69f-ff17-432a-a0ce-0c2a87601858" containerName="util" Jul 08 17:43:52.571931 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.571914 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51d3d69f-ff17-432a-a0ce-0c2a87601858" containerName="pull" Jul 08 17:43:52.571931 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.571920 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="51d3d69f-ff17-432a-a0ce-0c2a87601858" containerName="pull" Jul 08 17:43:52.572083 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.571973 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="51d3d69f-ff17-432a-a0ce-0c2a87601858" containerName="extract" Jul 08 17:43:52.574845 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.574823 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" Jul 08 17:43:52.577382 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.577359 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 08 17:43:52.577382 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.577378 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 17:43:52.577514 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.577405 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 17:43:52.577646 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.577632 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 17:43:52.577855 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.577839 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5jqb9\"" Jul 08 17:43:52.589330 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.589304 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt"] Jul 08 17:43:52.660434 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.660401 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1236d4aa-7255-4ce5-ac73-1bf9bb53806a-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dcf78d495-42ltt\" (UID: \"1236d4aa-7255-4ce5-ac73-1bf9bb53806a\") " pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" Jul 08 17:43:52.660614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.660462 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1236d4aa-7255-4ce5-ac73-1bf9bb53806a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dcf78d495-42ltt\" (UID: \"1236d4aa-7255-4ce5-ac73-1bf9bb53806a\") " pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" Jul 08 17:43:52.660614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.660487 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1236d4aa-7255-4ce5-ac73-1bf9bb53806a-tmp\") pod \"opendatahub-operator-controller-manager-6dcf78d495-42ltt\" (UID: \"1236d4aa-7255-4ce5-ac73-1bf9bb53806a\") " pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" Jul 08 17:43:52.660700 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.660630 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pc54\" (UniqueName: \"kubernetes.io/projected/1236d4aa-7255-4ce5-ac73-1bf9bb53806a-kube-api-access-2pc54\") pod \"opendatahub-operator-controller-manager-6dcf78d495-42ltt\" (UID: \"1236d4aa-7255-4ce5-ac73-1bf9bb53806a\") " pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" Jul 08 17:43:52.660700 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.660674 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1236d4aa-7255-4ce5-ac73-1bf9bb53806a-manifests\") pod \"opendatahub-operator-controller-manager-6dcf78d495-42ltt\" (UID: \"1236d4aa-7255-4ce5-ac73-1bf9bb53806a\") " pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" Jul 08 17:43:52.761232 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.761194 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pc54\" (UniqueName: \"kubernetes.io/projected/1236d4aa-7255-4ce5-ac73-1bf9bb53806a-kube-api-access-2pc54\") pod \"opendatahub-operator-controller-manager-6dcf78d495-42ltt\" (UID: \"1236d4aa-7255-4ce5-ac73-1bf9bb53806a\") " pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" Jul 08 17:43:52.761460 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.761232 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1236d4aa-7255-4ce5-ac73-1bf9bb53806a-manifests\") pod \"opendatahub-operator-controller-manager-6dcf78d495-42ltt\" (UID: \"1236d4aa-7255-4ce5-ac73-1bf9bb53806a\") " pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" Jul 08 17:43:52.761460 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.761298 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1236d4aa-7255-4ce5-ac73-1bf9bb53806a-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dcf78d495-42ltt\" (UID: \"1236d4aa-7255-4ce5-ac73-1bf9bb53806a\") " pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" Jul 08 17:43:52.761460 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.761330 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1236d4aa-7255-4ce5-ac73-1bf9bb53806a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dcf78d495-42ltt\" (UID: \"1236d4aa-7255-4ce5-ac73-1bf9bb53806a\") " pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" Jul 08 17:43:52.761460 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.761349 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1236d4aa-7255-4ce5-ac73-1bf9bb53806a-tmp\") pod \"opendatahub-operator-controller-manager-6dcf78d495-42ltt\" (UID: \"1236d4aa-7255-4ce5-ac73-1bf9bb53806a\") " pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" Jul 08 17:43:52.761701 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.761662 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1236d4aa-7255-4ce5-ac73-1bf9bb53806a-manifests\") pod \"opendatahub-operator-controller-manager-6dcf78d495-42ltt\" (UID: \"1236d4aa-7255-4ce5-ac73-1bf9bb53806a\") " pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" Jul 08 17:43:52.763960 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.763933 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1236d4aa-7255-4ce5-ac73-1bf9bb53806a-tmp\") pod \"opendatahub-operator-controller-manager-6dcf78d495-42ltt\" (UID: \"1236d4aa-7255-4ce5-ac73-1bf9bb53806a\") " pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" Jul 08 17:43:52.764075 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.763940 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1236d4aa-7255-4ce5-ac73-1bf9bb53806a-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dcf78d495-42ltt\" (UID: \"1236d4aa-7255-4ce5-ac73-1bf9bb53806a\") " pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" Jul 08 17:43:52.764319 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.764299 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1236d4aa-7255-4ce5-ac73-1bf9bb53806a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dcf78d495-42ltt\" (UID: \"1236d4aa-7255-4ce5-ac73-1bf9bb53806a\") " pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" Jul 08 17:43:52.772407 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.772383 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pc54\" (UniqueName: \"kubernetes.io/projected/1236d4aa-7255-4ce5-ac73-1bf9bb53806a-kube-api-access-2pc54\") pod \"opendatahub-operator-controller-manager-6dcf78d495-42ltt\" (UID: \"1236d4aa-7255-4ce5-ac73-1bf9bb53806a\") " pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" Jul 08 17:43:52.777075 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.777037 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:52.877741 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.877700 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:52.884420 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.884389 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" Jul 08 17:43:52.978298 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:52.978215 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:53.024601 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:53.024574 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt"] Jul 08 17:43:53.026443 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:43:53.026421 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1236d4aa_7255_4ce5_ac73_1bf9bb53806a.slice/crio-8a687026236db79b130154a4d3f123488df8740358932ece76f742b67091873e WatchSource:0}: Error finding container 8a687026236db79b130154a4d3f123488df8740358932ece76f742b67091873e: Status 404 returned error can't find the container with id 8a687026236db79b130154a4d3f123488df8740358932ece76f742b67091873e Jul 08 17:43:53.028360 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:53.028342 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 17:43:53.040686 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:53.040652 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:53.110430 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:53.110371 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:53.207433 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:53.207350 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:53.273875 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:53.273838 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" event={"ID":"1236d4aa-7255-4ce5-ac73-1bf9bb53806a","Type":"ContainerStarted","Data":"8a687026236db79b130154a4d3f123488df8740358932ece76f742b67091873e"} Jul 08 17:43:53.307780 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:53.307746 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:53.409887 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:53.409848 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:43:56.290261 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:56.290140 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" event={"ID":"1236d4aa-7255-4ce5-ac73-1bf9bb53806a","Type":"ContainerStarted","Data":"e9552ee982defd38e69b54f6e23d2d8030d51fe4dfa01eb2101ad047a9033b3b"} Jul 08 17:43:57.295770 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:57.295682 2584 generic.go:356] "Generic (PLEG): container finished" podID="1236d4aa-7255-4ce5-ac73-1bf9bb53806a" containerID="e9552ee982defd38e69b54f6e23d2d8030d51fe4dfa01eb2101ad047a9033b3b" exitCode=0 Jul 08 17:43:57.295770 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:57.295716 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" event={"ID":"1236d4aa-7255-4ce5-ac73-1bf9bb53806a","Type":"ContainerDied","Data":"e9552ee982defd38e69b54f6e23d2d8030d51fe4dfa01eb2101ad047a9033b3b"} Jul 08 17:43:58.303020 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:58.302980 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" event={"ID":"1236d4aa-7255-4ce5-ac73-1bf9bb53806a","Type":"ContainerStarted","Data":"6984ac29293ec9ba695c973996aa69726519433879bf3e6f37619f0be22e64d9"} Jul 08 17:43:58.305033 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:58.305006 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" Jul 08 17:43:58.330051 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:43:58.330002 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" podStartSLOduration=1.998036184 podStartE2EDuration="6.329988769s" totalImagesPullingTime="3.311706739s" totalInitContainerRuntime="1.020245846s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:43:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:43:58.328543272 +0000 UTC m=+314.012640548" watchObservedRunningTime="2026-07-08 17:43:58.329988769 +0000 UTC m=+314.014086045" Jul 08 17:44:03.763217 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:03.763169 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:03.872692 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:03.872646 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:10.315095 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:10.315064 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6dcf78d495-42ltt" Jul 08 17:44:12.845510 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:12.845467 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:12.964976 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:12.964939 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:13.434691 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.434654 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:13.500862 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.500824 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-64959b45fc-pdgnb"] Jul 08 17:44:13.505466 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.505445 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64959b45fc-pdgnb" Jul 08 17:44:13.507878 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.507856 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 08 17:44:13.509296 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.509275 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 08 17:44:13.519123 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.519101 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64959b45fc-pdgnb"] Jul 08 17:44:13.634354 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.634314 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7610607e-f0be-48c3-bdac-57396e5fb33e-tls-certs\") pod \"kube-auth-proxy-64959b45fc-pdgnb\" (UID: \"7610607e-f0be-48c3-bdac-57396e5fb33e\") " pod="openshift-ingress/kube-auth-proxy-64959b45fc-pdgnb" Jul 08 17:44:13.634560 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.634392 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tww4w\" (UniqueName: \"kubernetes.io/projected/7610607e-f0be-48c3-bdac-57396e5fb33e-kube-api-access-tww4w\") pod \"kube-auth-proxy-64959b45fc-pdgnb\" (UID: \"7610607e-f0be-48c3-bdac-57396e5fb33e\") " pod="openshift-ingress/kube-auth-proxy-64959b45fc-pdgnb" Jul 08 17:44:13.634560 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.634465 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7610607e-f0be-48c3-bdac-57396e5fb33e-tmp\") pod \"kube-auth-proxy-64959b45fc-pdgnb\" (UID: \"7610607e-f0be-48c3-bdac-57396e5fb33e\") " pod="openshift-ingress/kube-auth-proxy-64959b45fc-pdgnb" Jul 08 17:44:13.649632 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.649597 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-btzq8"] Jul 08 17:44:13.653187 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.653170 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-btzq8" Jul 08 17:44:13.655586 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.655565 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-gl2wv\"" Jul 08 17:44:13.655792 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.655780 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 08 17:44:13.665892 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.665863 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-btzq8"] Jul 08 17:44:13.691918 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.691815 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:13.736068 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.736032 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7610607e-f0be-48c3-bdac-57396e5fb33e-tls-certs\") pod \"kube-auth-proxy-64959b45fc-pdgnb\" (UID: \"7610607e-f0be-48c3-bdac-57396e5fb33e\") " pod="openshift-ingress/kube-auth-proxy-64959b45fc-pdgnb" Jul 08 17:44:13.736303 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.736084 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tww4w\" (UniqueName: \"kubernetes.io/projected/7610607e-f0be-48c3-bdac-57396e5fb33e-kube-api-access-tww4w\") pod \"kube-auth-proxy-64959b45fc-pdgnb\" (UID: \"7610607e-f0be-48c3-bdac-57396e5fb33e\") " pod="openshift-ingress/kube-auth-proxy-64959b45fc-pdgnb" Jul 08 17:44:13.736303 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.736108 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7610607e-f0be-48c3-bdac-57396e5fb33e-tmp\") pod \"kube-auth-proxy-64959b45fc-pdgnb\" (UID: \"7610607e-f0be-48c3-bdac-57396e5fb33e\") " pod="openshift-ingress/kube-auth-proxy-64959b45fc-pdgnb" Jul 08 17:44:13.736303 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:44:13.736195 2584 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 08 17:44:13.736477 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:44:13.736314 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7610607e-f0be-48c3-bdac-57396e5fb33e-tls-certs podName:7610607e-f0be-48c3-bdac-57396e5fb33e nodeName:}" failed. No retries permitted until 2026-07-08 17:44:14.236290292 +0000 UTC m=+329.920387561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7610607e-f0be-48c3-bdac-57396e5fb33e-tls-certs") pod "kube-auth-proxy-64959b45fc-pdgnb" (UID: "7610607e-f0be-48c3-bdac-57396e5fb33e") : secret "kube-auth-proxy-tls" not found Jul 08 17:44:13.738529 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.738479 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7610607e-f0be-48c3-bdac-57396e5fb33e-tmp\") pod \"kube-auth-proxy-64959b45fc-pdgnb\" (UID: \"7610607e-f0be-48c3-bdac-57396e5fb33e\") " pod="openshift-ingress/kube-auth-proxy-64959b45fc-pdgnb" Jul 08 17:44:13.747538 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.747514 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tww4w\" (UniqueName: \"kubernetes.io/projected/7610607e-f0be-48c3-bdac-57396e5fb33e-kube-api-access-tww4w\") pod \"kube-auth-proxy-64959b45fc-pdgnb\" (UID: \"7610607e-f0be-48c3-bdac-57396e5fb33e\") " pod="openshift-ingress/kube-auth-proxy-64959b45fc-pdgnb" Jul 08 17:44:13.776626 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.776586 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:13.837377 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.837340 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53eb9f89-11ea-48d4-9c9c-6720d0ae0950-cert\") pod \"odh-model-controller-7998c67dc4-btzq8\" (UID: \"53eb9f89-11ea-48d4-9c9c-6720d0ae0950\") " pod="opendatahub/odh-model-controller-7998c67dc4-btzq8" Jul 08 17:44:13.837549 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.837439 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jfb7\" (UniqueName: \"kubernetes.io/projected/53eb9f89-11ea-48d4-9c9c-6720d0ae0950-kube-api-access-7jfb7\") pod \"odh-model-controller-7998c67dc4-btzq8\" (UID: \"53eb9f89-11ea-48d4-9c9c-6720d0ae0950\") " pod="opendatahub/odh-model-controller-7998c67dc4-btzq8" Jul 08 17:44:13.877663 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.877616 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:13.937884 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.937836 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jfb7\" (UniqueName: \"kubernetes.io/projected/53eb9f89-11ea-48d4-9c9c-6720d0ae0950-kube-api-access-7jfb7\") pod \"odh-model-controller-7998c67dc4-btzq8\" (UID: \"53eb9f89-11ea-48d4-9c9c-6720d0ae0950\") " pod="opendatahub/odh-model-controller-7998c67dc4-btzq8" Jul 08 17:44:13.938087 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.937924 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53eb9f89-11ea-48d4-9c9c-6720d0ae0950-cert\") pod \"odh-model-controller-7998c67dc4-btzq8\" (UID: \"53eb9f89-11ea-48d4-9c9c-6720d0ae0950\") " pod="opendatahub/odh-model-controller-7998c67dc4-btzq8" Jul 08 17:44:13.940413 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.940387 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/53eb9f89-11ea-48d4-9c9c-6720d0ae0950-cert\") pod \"odh-model-controller-7998c67dc4-btzq8\" (UID: \"53eb9f89-11ea-48d4-9c9c-6720d0ae0950\") " pod="opendatahub/odh-model-controller-7998c67dc4-btzq8" Jul 08 17:44:13.948072 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.948012 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jfb7\" (UniqueName: \"kubernetes.io/projected/53eb9f89-11ea-48d4-9c9c-6720d0ae0950-kube-api-access-7jfb7\") pod \"odh-model-controller-7998c67dc4-btzq8\" (UID: \"53eb9f89-11ea-48d4-9c9c-6720d0ae0950\") " pod="opendatahub/odh-model-controller-7998c67dc4-btzq8" Jul 08 17:44:13.964394 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.964365 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-btzq8" Jul 08 17:44:13.969150 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:13.969117 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:14.114639 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.114614 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-btzq8"] Jul 08 17:44:14.116576 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:44:14.116548 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53eb9f89_11ea_48d4_9c9c_6720d0ae0950.slice/crio-02605b11bd8b8ce86c506e7bcefb301814325615dc6379988f8678c27da1a88e WatchSource:0}: Error finding container 02605b11bd8b8ce86c506e7bcefb301814325615dc6379988f8678c27da1a88e: Status 404 returned error can't find the container with id 02605b11bd8b8ce86c506e7bcefb301814325615dc6379988f8678c27da1a88e Jul 08 17:44:14.240861 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.240757 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7610607e-f0be-48c3-bdac-57396e5fb33e-tls-certs\") pod \"kube-auth-proxy-64959b45fc-pdgnb\" (UID: \"7610607e-f0be-48c3-bdac-57396e5fb33e\") " pod="openshift-ingress/kube-auth-proxy-64959b45fc-pdgnb" Jul 08 17:44:14.243172 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.243143 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7610607e-f0be-48c3-bdac-57396e5fb33e-tls-certs\") pod \"kube-auth-proxy-64959b45fc-pdgnb\" (UID: \"7610607e-f0be-48c3-bdac-57396e5fb33e\") " pod="openshift-ingress/kube-auth-proxy-64959b45fc-pdgnb" Jul 08 17:44:14.373449 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.373409 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-btzq8" event={"ID":"53eb9f89-11ea-48d4-9c9c-6720d0ae0950","Type":"ContainerStarted","Data":"02605b11bd8b8ce86c506e7bcefb301814325615dc6379988f8678c27da1a88e"} Jul 08 17:44:14.414593 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.414560 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-64959b45fc-pdgnb" Jul 08 17:44:14.575526 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.575361 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:14.575617 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.575591 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-64959b45fc-pdgnb"] Jul 08 17:44:14.578430 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:44:14.578399 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7610607e_f0be_48c3_bdac_57396e5fb33e.slice/crio-c4bddd9b256ce669bc92b8104445b28750a556e416fc656932dd7d0234dc4116 WatchSource:0}: Error finding container c4bddd9b256ce669bc92b8104445b28750a556e416fc656932dd7d0234dc4116: Status 404 returned error can't find the container with id c4bddd9b256ce669bc92b8104445b28750a556e416fc656932dd7d0234dc4116 Jul 08 17:44:14.604160 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.604122 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vcspn"] Jul 08 17:44:14.608963 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.608939 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vcspn" Jul 08 17:44:14.611471 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.611442 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 08 17:44:14.611580 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.611486 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-kfx7m\"" Jul 08 17:44:14.618024 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.617998 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vcspn"] Jul 08 17:44:14.745551 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.745512 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be30b1e-b173-44b1-a2d3-39cd8183b9cc-tls-certs\") pod \"model-serving-api-54d89bd5fc-vcspn\" (UID: \"6be30b1e-b173-44b1-a2d3-39cd8183b9cc\") " pod="opendatahub/model-serving-api-54d89bd5fc-vcspn" Jul 08 17:44:14.745699 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.745566 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-242g8\" (UniqueName: \"kubernetes.io/projected/6be30b1e-b173-44b1-a2d3-39cd8183b9cc-kube-api-access-242g8\") pod \"model-serving-api-54d89bd5fc-vcspn\" (UID: \"6be30b1e-b173-44b1-a2d3-39cd8183b9cc\") " pod="opendatahub/model-serving-api-54d89bd5fc-vcspn" Jul 08 17:44:14.847024 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.846916 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be30b1e-b173-44b1-a2d3-39cd8183b9cc-tls-certs\") pod \"model-serving-api-54d89bd5fc-vcspn\" (UID: \"6be30b1e-b173-44b1-a2d3-39cd8183b9cc\") " pod="opendatahub/model-serving-api-54d89bd5fc-vcspn" Jul 08 17:44:14.847024 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.846962 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-242g8\" (UniqueName: \"kubernetes.io/projected/6be30b1e-b173-44b1-a2d3-39cd8183b9cc-kube-api-access-242g8\") pod \"model-serving-api-54d89bd5fc-vcspn\" (UID: \"6be30b1e-b173-44b1-a2d3-39cd8183b9cc\") " pod="opendatahub/model-serving-api-54d89bd5fc-vcspn" Jul 08 17:44:14.847293 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:44:14.847076 2584 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 08 17:44:14.847293 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:44:14.847153 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6be30b1e-b173-44b1-a2d3-39cd8183b9cc-tls-certs podName:6be30b1e-b173-44b1-a2d3-39cd8183b9cc nodeName:}" failed. No retries permitted until 2026-07-08 17:44:15.347135362 +0000 UTC m=+331.031232617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6be30b1e-b173-44b1-a2d3-39cd8183b9cc-tls-certs") pod "model-serving-api-54d89bd5fc-vcspn" (UID: "6be30b1e-b173-44b1-a2d3-39cd8183b9cc") : secret "model-serving-api-tls" not found Jul 08 17:44:14.849743 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.849708 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:14.856745 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.856709 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-k29q6"] Jul 08 17:44:14.860494 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.860473 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:14.862876 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.862846 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-242g8\" (UniqueName: \"kubernetes.io/projected/6be30b1e-b173-44b1-a2d3-39cd8183b9cc-kube-api-access-242g8\") pod \"model-serving-api-54d89bd5fc-vcspn\" (UID: \"6be30b1e-b173-44b1-a2d3-39cd8183b9cc\") " pod="opendatahub/model-serving-api-54d89bd5fc-vcspn" Jul 08 17:44:14.863020 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.862879 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 08 17:44:14.863357 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.863325 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 08 17:44:14.863357 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.863336 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-shwpq\"" Jul 08 17:44:14.863357 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.863325 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 08 17:44:14.863562 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.863365 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 17:44:14.874847 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.874815 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-k29q6"] Jul 08 17:44:14.922505 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:14.922467 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:15.049421 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.049386 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b165f2-6057-4d01-8610-3fae7e9b874f-local-certs\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.049585 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.049431 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.049585 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.049537 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/24b165f2-6057-4d01-8610-3fae7e9b874f-cacerts\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.049585 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.049560 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.049721 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.049591 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-token\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.049721 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.049635 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgtsm\" (UniqueName: \"kubernetes.io/projected/24b165f2-6057-4d01-8610-3fae7e9b874f-kube-api-access-rgtsm\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.049721 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.049666 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.086734 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.086700 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:15.150984 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.150947 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.151201 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.151006 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b165f2-6057-4d01-8610-3fae7e9b874f-local-certs\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.151201 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.151039 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.151201 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.151112 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/24b165f2-6057-4d01-8610-3fae7e9b874f-cacerts\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.151201 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.151138 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.151201 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.151170 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-token\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.151474 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.151205 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgtsm\" (UniqueName: \"kubernetes.io/projected/24b165f2-6057-4d01-8610-3fae7e9b874f-kube-api-access-rgtsm\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.151710 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.151679 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.162256 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.154401 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.162256 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.154483 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/24b165f2-6057-4d01-8610-3fae7e9b874f-cacerts\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.162256 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.154558 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b165f2-6057-4d01-8610-3fae7e9b874f-local-certs\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.162256 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.154602 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.163742 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.163710 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-token\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.164814 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.164786 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgtsm\" (UniqueName: \"kubernetes.io/projected/24b165f2-6057-4d01-8610-3fae7e9b874f-kube-api-access-rgtsm\") pod \"istiod-openshift-gateway-7574459656-k29q6\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.179255 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.179201 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:15.330236 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.330196 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-k29q6"] Jul 08 17:44:15.331835 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:44:15.331798 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b165f2_6057_4d01_8610_3fae7e9b874f.slice/crio-45fa91a240597ee226cb5e973bdcbefb0a5e15d9ad2ae2d72a4b997a0d9a853c WatchSource:0}: Error finding container 45fa91a240597ee226cb5e973bdcbefb0a5e15d9ad2ae2d72a4b997a0d9a853c: Status 404 returned error can't find the container with id 45fa91a240597ee226cb5e973bdcbefb0a5e15d9ad2ae2d72a4b997a0d9a853c Jul 08 17:44:15.352795 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.352702 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be30b1e-b173-44b1-a2d3-39cd8183b9cc-tls-certs\") pod \"model-serving-api-54d89bd5fc-vcspn\" (UID: \"6be30b1e-b173-44b1-a2d3-39cd8183b9cc\") " pod="opendatahub/model-serving-api-54d89bd5fc-vcspn" Jul 08 17:44:15.356740 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.356688 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be30b1e-b173-44b1-a2d3-39cd8183b9cc-tls-certs\") pod \"model-serving-api-54d89bd5fc-vcspn\" (UID: \"6be30b1e-b173-44b1-a2d3-39cd8183b9cc\") " pod="opendatahub/model-serving-api-54d89bd5fc-vcspn" Jul 08 17:44:15.381348 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.381230 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64959b45fc-pdgnb" event={"ID":"7610607e-f0be-48c3-bdac-57396e5fb33e","Type":"ContainerStarted","Data":"c4bddd9b256ce669bc92b8104445b28750a556e416fc656932dd7d0234dc4116"} Jul 08 17:44:15.384719 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.384681 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" event={"ID":"24b165f2-6057-4d01-8610-3fae7e9b874f","Type":"ContainerStarted","Data":"45fa91a240597ee226cb5e973bdcbefb0a5e15d9ad2ae2d72a4b997a0d9a853c"} Jul 08 17:44:15.520565 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.520433 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-vcspn" Jul 08 17:44:15.708612 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:15.708580 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-vcspn"] Jul 08 17:44:15.712690 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:44:15.712657 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be30b1e_b173_44b1_a2d3_39cd8183b9cc.slice/crio-50b828d7f1549eb039ba67b523b53f44b7ec08d3c22a4a16b052db0f4427d411 WatchSource:0}: Error finding container 50b828d7f1549eb039ba67b523b53f44b7ec08d3c22a4a16b052db0f4427d411: Status 404 returned error can't find the container with id 50b828d7f1549eb039ba67b523b53f44b7ec08d3c22a4a16b052db0f4427d411 Jul 08 17:44:16.148556 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:16.148464 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:16.393953 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:16.393912 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vcspn" event={"ID":"6be30b1e-b173-44b1-a2d3-39cd8183b9cc","Type":"ContainerStarted","Data":"50b828d7f1549eb039ba67b523b53f44b7ec08d3c22a4a16b052db0f4427d411"} Jul 08 17:44:20.280009 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:20.279961 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 17:44:20.280292 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:20.280058 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 17:44:20.420326 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:20.420282 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-64959b45fc-pdgnb" event={"ID":"7610607e-f0be-48c3-bdac-57396e5fb33e","Type":"ContainerStarted","Data":"eaa760a5d542da01f513d9677a830fc0c26a2d3b052db88f2efbec4a6e4661a8"} Jul 08 17:44:20.422529 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:20.422495 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" event={"ID":"24b165f2-6057-4d01-8610-3fae7e9b874f","Type":"ContainerStarted","Data":"a8606d7db4a29cea7bf21b74bc74da8f25c3ade775456789c2bd7043dfe75c8f"} Jul 08 17:44:20.423097 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:20.423070 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:20.424885 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:20.424853 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-vcspn" event={"ID":"6be30b1e-b173-44b1-a2d3-39cd8183b9cc","Type":"ContainerStarted","Data":"824b24e8b1fd3b2643cd0edc81ccb120410c47faccb9496196290bd23a85ca0e"} Jul 08 17:44:20.424984 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:20.424919 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-vcspn" Jul 08 17:44:20.427010 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:20.426982 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-btzq8" event={"ID":"53eb9f89-11ea-48d4-9c9c-6720d0ae0950","Type":"ContainerStarted","Data":"98c4267e87c9530cfb3006ee30cdf203b8ad780cc74aa3a47b7cb0065a08342b"} Jul 08 17:44:20.427121 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:20.427045 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-btzq8" Jul 08 17:44:20.440428 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:20.440377 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-64959b45fc-pdgnb" podStartSLOduration=1.744046553 podStartE2EDuration="7.440362582s" totalImagesPullingTime="5.696316029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:44:20.438157614 +0000 UTC m=+336.122254892" watchObservedRunningTime="2026-07-08 17:44:20.440362582 +0000 UTC m=+336.124459903" Jul 08 17:44:20.460643 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:20.460580 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-btzq8" podStartSLOduration=1.293119259 podStartE2EDuration="7.460559831s" totalImagesPullingTime="6.167440572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:44:20.458765336 +0000 UTC m=+336.142862614" watchObservedRunningTime="2026-07-08 17:44:20.460559831 +0000 UTC m=+336.144657108" Jul 08 17:44:20.482823 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:20.482731 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" podStartSLOduration=1.5372391319999998 podStartE2EDuration="6.482706939s" totalImagesPullingTime="4.945467807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:44:20.481566419 +0000 UTC m=+336.165663696" watchObservedRunningTime="2026-07-08 17:44:20.482706939 +0000 UTC m=+336.166804216" Jul 08 17:44:20.501805 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:20.501707 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-vcspn" podStartSLOduration=1.934707534 podStartE2EDuration="6.501693054s" totalImagesPullingTime="4.56698552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:44:20.499294596 +0000 UTC m=+336.183391875" watchObservedRunningTime="2026-07-08 17:44:20.501693054 +0000 UTC m=+336.185790330" Jul 08 17:44:21.093772 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:21.093733 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:21.191219 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:21.191184 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:21.434230 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:21.434196 2584 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-k29q6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 08 17:44:21.434594 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:21.434277 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" podUID="24b165f2-6057-4d01-8610-3fae7e9b874f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 17:44:22.001035 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:22.000996 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:22.333918 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:22.333829 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:22.439062 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:22.439027 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:22.648565 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:22.648514 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:25.955365 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:25.955291 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:26.033693 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.033652 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:26.034275 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.034252 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-k56cv"] Jul 08 17:44:26.040618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.040596 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k56cv" Jul 08 17:44:26.043150 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.043124 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 08 17:44:26.043283 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.043216 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-z9j64\"" Jul 08 17:44:26.048348 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.048322 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6fkzv"] Jul 08 17:44:26.052507 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.052485 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6fkzv" Jul 08 17:44:26.052920 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.052892 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-k56cv"] Jul 08 17:44:26.054878 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.054856 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 08 17:44:26.054969 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.054890 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-nfqvr\"" Jul 08 17:44:26.063988 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.063955 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6fkzv"] Jul 08 17:44:26.106443 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.106404 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:26.160218 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.160179 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bshp\" (UniqueName: \"kubernetes.io/projected/8bc4bc8b-6633-48d4-942d-3eae3cca72b8-kube-api-access-7bshp\") pod \"kserve-controller-manager-d6f4b65b8-k56cv\" (UID: \"8bc4bc8b-6633-48d4-942d-3eae3cca72b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k56cv" Jul 08 17:44:26.160218 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.160223 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aba62f91-4709-4884-8435-74012fe550ea-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6fkzv\" (UID: \"aba62f91-4709-4884-8435-74012fe550ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6fkzv" Jul 08 17:44:26.160456 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.160337 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn4cw\" (UniqueName: \"kubernetes.io/projected/aba62f91-4709-4884-8435-74012fe550ea-kube-api-access-jn4cw\") pod \"llmisvc-controller-manager-777c6bd7d5-6fkzv\" (UID: \"aba62f91-4709-4884-8435-74012fe550ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6fkzv" Jul 08 17:44:26.160456 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.160369 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bc4bc8b-6633-48d4-942d-3eae3cca72b8-cert\") pod \"kserve-controller-manager-d6f4b65b8-k56cv\" (UID: \"8bc4bc8b-6633-48d4-942d-3eae3cca72b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k56cv" Jul 08 17:44:26.261798 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.261705 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bc4bc8b-6633-48d4-942d-3eae3cca72b8-cert\") pod \"kserve-controller-manager-d6f4b65b8-k56cv\" (UID: \"8bc4bc8b-6633-48d4-942d-3eae3cca72b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k56cv" Jul 08 17:44:26.261798 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.261758 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bshp\" (UniqueName: \"kubernetes.io/projected/8bc4bc8b-6633-48d4-942d-3eae3cca72b8-kube-api-access-7bshp\") pod \"kserve-controller-manager-d6f4b65b8-k56cv\" (UID: \"8bc4bc8b-6633-48d4-942d-3eae3cca72b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k56cv" Jul 08 17:44:26.261798 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.261781 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aba62f91-4709-4884-8435-74012fe550ea-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6fkzv\" (UID: \"aba62f91-4709-4884-8435-74012fe550ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6fkzv" Jul 08 17:44:26.262144 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.261843 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn4cw\" (UniqueName: \"kubernetes.io/projected/aba62f91-4709-4884-8435-74012fe550ea-kube-api-access-jn4cw\") pod \"llmisvc-controller-manager-777c6bd7d5-6fkzv\" (UID: \"aba62f91-4709-4884-8435-74012fe550ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6fkzv" Jul 08 17:44:26.262144 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:44:26.261942 2584 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 08 17:44:26.262144 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:44:26.262006 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba62f91-4709-4884-8435-74012fe550ea-cert podName:aba62f91-4709-4884-8435-74012fe550ea nodeName:}" failed. No retries permitted until 2026-07-08 17:44:26.761985826 +0000 UTC m=+342.446083084 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/aba62f91-4709-4884-8435-74012fe550ea-cert") pod "llmisvc-controller-manager-777c6bd7d5-6fkzv" (UID: "aba62f91-4709-4884-8435-74012fe550ea") : secret "llmisvc-webhook-server-cert" not found Jul 08 17:44:26.264356 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.264332 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bc4bc8b-6633-48d4-942d-3eae3cca72b8-cert\") pod \"kserve-controller-manager-d6f4b65b8-k56cv\" (UID: \"8bc4bc8b-6633-48d4-942d-3eae3cca72b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k56cv" Jul 08 17:44:26.273075 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.273048 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn4cw\" (UniqueName: \"kubernetes.io/projected/aba62f91-4709-4884-8435-74012fe550ea-kube-api-access-jn4cw\") pod \"llmisvc-controller-manager-777c6bd7d5-6fkzv\" (UID: \"aba62f91-4709-4884-8435-74012fe550ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6fkzv" Jul 08 17:44:26.273438 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.273419 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bshp\" (UniqueName: \"kubernetes.io/projected/8bc4bc8b-6633-48d4-942d-3eae3cca72b8-kube-api-access-7bshp\") pod \"kserve-controller-manager-d6f4b65b8-k56cv\" (UID: \"8bc4bc8b-6633-48d4-942d-3eae3cca72b8\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k56cv" Jul 08 17:44:26.352489 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.352451 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k56cv" Jul 08 17:44:26.517801 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.517724 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-k56cv"] Jul 08 17:44:26.520890 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:44:26.520863 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc4bc8b_6633_48d4_942d_3eae3cca72b8.slice/crio-36e869274f73c51ed01ecfaa383d3af7e866681a68dba06f10090c7396c2c38c WatchSource:0}: Error finding container 36e869274f73c51ed01ecfaa383d3af7e866681a68dba06f10090c7396c2c38c: Status 404 returned error can't find the container with id 36e869274f73c51ed01ecfaa383d3af7e866681a68dba06f10090c7396c2c38c Jul 08 17:44:26.523289 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.523215 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:26.767257 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.767212 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aba62f91-4709-4884-8435-74012fe550ea-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6fkzv\" (UID: \"aba62f91-4709-4884-8435-74012fe550ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6fkzv" Jul 08 17:44:26.767410 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:44:26.767350 2584 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 08 17:44:26.767451 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:44:26.767424 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aba62f91-4709-4884-8435-74012fe550ea-cert podName:aba62f91-4709-4884-8435-74012fe550ea nodeName:}" failed. No retries permitted until 2026-07-08 17:44:27.7674078 +0000 UTC m=+343.451505059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/aba62f91-4709-4884-8435-74012fe550ea-cert") pod "llmisvc-controller-manager-777c6bd7d5-6fkzv" (UID: "aba62f91-4709-4884-8435-74012fe550ea") : secret "llmisvc-webhook-server-cert" not found Jul 08 17:44:26.886331 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:26.886298 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:27.265633 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:27.265549 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:27.335032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:27.334998 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:27.404131 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:27.404095 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:44:27.468751 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:27.468713 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k56cv" event={"ID":"8bc4bc8b-6633-48d4-942d-3eae3cca72b8","Type":"ContainerStarted","Data":"36e869274f73c51ed01ecfaa383d3af7e866681a68dba06f10090c7396c2c38c"} Jul 08 17:44:27.776358 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:27.776314 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aba62f91-4709-4884-8435-74012fe550ea-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6fkzv\" (UID: \"aba62f91-4709-4884-8435-74012fe550ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6fkzv" Jul 08 17:44:27.778729 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:27.778698 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aba62f91-4709-4884-8435-74012fe550ea-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-6fkzv\" (UID: \"aba62f91-4709-4884-8435-74012fe550ea\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6fkzv" Jul 08 17:44:27.864828 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:27.864796 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6fkzv" Jul 08 17:44:28.012040 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:28.012013 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-6fkzv"] Jul 08 17:44:28.013915 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:44:28.013885 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaba62f91_4709_4884_8435_74012fe550ea.slice/crio-faf87a8a31c99665b1b27023cfb5fb75e027569ecc7dc09516a1ae10dcefd16a WatchSource:0}: Error finding container faf87a8a31c99665b1b27023cfb5fb75e027569ecc7dc09516a1ae10dcefd16a: Status 404 returned error can't find the container with id faf87a8a31c99665b1b27023cfb5fb75e027569ecc7dc09516a1ae10dcefd16a Jul 08 17:44:28.475833 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:28.475799 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6fkzv" event={"ID":"aba62f91-4709-4884-8435-74012fe550ea","Type":"ContainerStarted","Data":"faf87a8a31c99665b1b27023cfb5fb75e027569ecc7dc09516a1ae10dcefd16a"} Jul 08 17:44:29.482971 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:29.482931 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k56cv" event={"ID":"8bc4bc8b-6633-48d4-942d-3eae3cca72b8","Type":"ContainerStarted","Data":"78ad112b2c487abc4adbcf0ba55ba8a1d7929f77a60334d99a8a403e88bdb322"} Jul 08 17:44:29.483426 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:29.483069 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k56cv" Jul 08 17:44:29.505890 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:29.505833 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k56cv" podStartSLOduration=0.946436017 podStartE2EDuration="3.50581349s" totalImagesPullingTime="2.559377473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:44:29.504425436 +0000 UTC m=+345.188522712" watchObservedRunningTime="2026-07-08 17:44:29.50581349 +0000 UTC m=+345.189910770" Jul 08 17:44:30.845744 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:30.845706 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-k29q6"] Jul 08 17:44:30.846173 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:30.845898 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" podUID="24b165f2-6057-4d01-8610-3fae7e9b874f" containerName="discovery" containerID="cri-o://a8606d7db4a29cea7bf21b74bc74da8f25c3ade775456789c2bd7043dfe75c8f" gracePeriod=30 Jul 08 17:44:31.108726 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.108664 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:31.209796 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.209763 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/24b165f2-6057-4d01-8610-3fae7e9b874f-cacerts\") pod \"24b165f2-6057-4d01-8610-3fae7e9b874f\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " Jul 08 17:44:31.209972 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.209809 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgtsm\" (UniqueName: \"kubernetes.io/projected/24b165f2-6057-4d01-8610-3fae7e9b874f-kube-api-access-rgtsm\") pod \"24b165f2-6057-4d01-8610-3fae7e9b874f\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " Jul 08 17:44:31.209972 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.209853 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-csr-ca-configmap\") pod \"24b165f2-6057-4d01-8610-3fae7e9b874f\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " Jul 08 17:44:31.209972 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.209876 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-token\") pod \"24b165f2-6057-4d01-8610-3fae7e9b874f\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " Jul 08 17:44:31.209972 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.209899 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b165f2-6057-4d01-8610-3fae7e9b874f-local-certs\") pod \"24b165f2-6057-4d01-8610-3fae7e9b874f\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " Jul 08 17:44:31.209972 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.209916 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-kubeconfig\") pod \"24b165f2-6057-4d01-8610-3fae7e9b874f\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " Jul 08 17:44:31.209972 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.209943 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-csr-dns-cert\") pod \"24b165f2-6057-4d01-8610-3fae7e9b874f\" (UID: \"24b165f2-6057-4d01-8610-3fae7e9b874f\") " Jul 08 17:44:31.210431 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.210371 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "24b165f2-6057-4d01-8610-3fae7e9b874f" (UID: "24b165f2-6057-4d01-8610-3fae7e9b874f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:44:31.212373 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.212339 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "24b165f2-6057-4d01-8610-3fae7e9b874f" (UID: "24b165f2-6057-4d01-8610-3fae7e9b874f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:44:31.212486 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.212381 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b165f2-6057-4d01-8610-3fae7e9b874f-kube-api-access-rgtsm" (OuterVolumeSpecName: "kube-api-access-rgtsm") pod "24b165f2-6057-4d01-8610-3fae7e9b874f" (UID: "24b165f2-6057-4d01-8610-3fae7e9b874f"). InnerVolumeSpecName "kube-api-access-rgtsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:44:31.212546 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.212494 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "24b165f2-6057-4d01-8610-3fae7e9b874f" (UID: "24b165f2-6057-4d01-8610-3fae7e9b874f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:44:31.212600 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.212551 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b165f2-6057-4d01-8610-3fae7e9b874f-cacerts" (OuterVolumeSpecName: "cacerts") pod "24b165f2-6057-4d01-8610-3fae7e9b874f" (UID: "24b165f2-6057-4d01-8610-3fae7e9b874f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:44:31.212600 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.212582 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b165f2-6057-4d01-8610-3fae7e9b874f-local-certs" (OuterVolumeSpecName: "local-certs") pod "24b165f2-6057-4d01-8610-3fae7e9b874f" (UID: "24b165f2-6057-4d01-8610-3fae7e9b874f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:44:31.212726 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.212702 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-token" (OuterVolumeSpecName: "istio-token") pod "24b165f2-6057-4d01-8610-3fae7e9b874f" (UID: "24b165f2-6057-4d01-8610-3fae7e9b874f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:44:31.311039 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.310999 2584 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/24b165f2-6057-4d01-8610-3fae7e9b874f-local-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:44:31.311039 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.311036 2584 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-kubeconfig\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:44:31.311276 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.311051 2584 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-csr-dns-cert\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:44:31.311276 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.311060 2584 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/24b165f2-6057-4d01-8610-3fae7e9b874f-cacerts\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:44:31.311276 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.311069 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgtsm\" (UniqueName: \"kubernetes.io/projected/24b165f2-6057-4d01-8610-3fae7e9b874f-kube-api-access-rgtsm\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:44:31.311276 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.311078 2584 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-csr-ca-configmap\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:44:31.311276 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.311086 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24b165f2-6057-4d01-8610-3fae7e9b874f-istio-token\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:44:31.435055 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.435022 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-btzq8" Jul 08 17:44:31.437258 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.437220 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-vcspn" Jul 08 17:44:31.497061 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.497022 2584 generic.go:356] "Generic (PLEG): container finished" podID="24b165f2-6057-4d01-8610-3fae7e9b874f" containerID="a8606d7db4a29cea7bf21b74bc74da8f25c3ade775456789c2bd7043dfe75c8f" exitCode=0 Jul 08 17:44:31.497275 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.497119 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" event={"ID":"24b165f2-6057-4d01-8610-3fae7e9b874f","Type":"ContainerDied","Data":"a8606d7db4a29cea7bf21b74bc74da8f25c3ade775456789c2bd7043dfe75c8f"} Jul 08 17:44:31.497275 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.497189 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" event={"ID":"24b165f2-6057-4d01-8610-3fae7e9b874f","Type":"ContainerDied","Data":"45fa91a240597ee226cb5e973bdcbefb0a5e15d9ad2ae2d72a4b997a0d9a853c"} Jul 08 17:44:31.497275 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.497218 2584 scope.go:117] "RemoveContainer" containerID="a8606d7db4a29cea7bf21b74bc74da8f25c3ade775456789c2bd7043dfe75c8f" Jul 08 17:44:31.497275 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.497219 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-k29q6" Jul 08 17:44:31.529114 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.529083 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-k29q6"] Jul 08 17:44:31.533940 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.533916 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-k29q6"] Jul 08 17:44:31.876963 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.876796 2584 scope.go:117] "RemoveContainer" containerID="a8606d7db4a29cea7bf21b74bc74da8f25c3ade775456789c2bd7043dfe75c8f" Jul 08 17:44:31.877283 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:44:31.877195 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8606d7db4a29cea7bf21b74bc74da8f25c3ade775456789c2bd7043dfe75c8f\": container with ID starting with a8606d7db4a29cea7bf21b74bc74da8f25c3ade775456789c2bd7043dfe75c8f not found: ID does not exist" containerID="a8606d7db4a29cea7bf21b74bc74da8f25c3ade775456789c2bd7043dfe75c8f" Jul 08 17:44:31.877283 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:31.877224 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8606d7db4a29cea7bf21b74bc74da8f25c3ade775456789c2bd7043dfe75c8f"} err="failed to get container status \"a8606d7db4a29cea7bf21b74bc74da8f25c3ade775456789c2bd7043dfe75c8f\": rpc error: code = NotFound desc = could not find container \"a8606d7db4a29cea7bf21b74bc74da8f25c3ade775456789c2bd7043dfe75c8f\": container with ID starting with a8606d7db4a29cea7bf21b74bc74da8f25c3ade775456789c2bd7043dfe75c8f not found: ID does not exist" Jul 08 17:44:32.504306 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:32.504273 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6fkzv" event={"ID":"aba62f91-4709-4884-8435-74012fe550ea","Type":"ContainerStarted","Data":"25873948f6e11dbdb93760f4687314ae20aea494824b7985005081e9c0340afb"} Jul 08 17:44:32.504484 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:32.504409 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6fkzv" Jul 08 17:44:32.528166 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:32.527988 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6fkzv" podStartSLOduration=2.6164485859999997 podStartE2EDuration="6.527969137s" totalImagesPullingTime="3.911520551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:44:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:44:32.527828199 +0000 UTC m=+348.211925489" watchObservedRunningTime="2026-07-08 17:44:32.527969137 +0000 UTC m=+348.212066414" Jul 08 17:44:32.958528 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:44:32.958488 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24b165f2-6057-4d01-8610-3fae7e9b874f" path="/var/lib/kubelet/pods/24b165f2-6057-4d01-8610-3fae7e9b874f/volumes" Jul 08 17:45:00.494282 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:00.494225 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k56cv" Jul 08 17:45:03.511553 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:03.511517 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-6fkzv" Jul 08 17:45:14.318072 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:14.318023 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:14.410729 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:14.410686 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:14.446661 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:14.446621 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dpx8r"] Jul 08 17:45:14.447005 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:14.446987 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24b165f2-6057-4d01-8610-3fae7e9b874f" containerName="discovery" Jul 08 17:45:14.447005 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:14.447002 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="24b165f2-6057-4d01-8610-3fae7e9b874f" containerName="discovery" Jul 08 17:45:14.447186 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:14.447064 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="24b165f2-6057-4d01-8610-3fae7e9b874f" containerName="discovery" Jul 08 17:45:14.450117 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:14.450099 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dpx8r" Jul 08 17:45:14.453297 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:14.453271 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-zlkfl\"" Jul 08 17:45:14.453485 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:14.453295 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 17:45:14.453565 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:14.453295 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 17:45:14.458985 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:14.458960 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dpx8r"] Jul 08 17:45:14.579359 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:14.579229 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw5p6\" (UniqueName: \"kubernetes.io/projected/77c86f3f-cd04-4cf2-84fe-a82ceaea50c6-kube-api-access-hw5p6\") pod \"kuadrant-operator-catalog-dpx8r\" (UID: \"77c86f3f-cd04-4cf2-84fe-a82ceaea50c6\") " pod="kuadrant-system/kuadrant-operator-catalog-dpx8r" Jul 08 17:45:14.680798 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:14.680708 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw5p6\" (UniqueName: \"kubernetes.io/projected/77c86f3f-cd04-4cf2-84fe-a82ceaea50c6-kube-api-access-hw5p6\") pod \"kuadrant-operator-catalog-dpx8r\" (UID: \"77c86f3f-cd04-4cf2-84fe-a82ceaea50c6\") " pod="kuadrant-system/kuadrant-operator-catalog-dpx8r" Jul 08 17:45:14.689943 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:14.689912 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw5p6\" (UniqueName: \"kubernetes.io/projected/77c86f3f-cd04-4cf2-84fe-a82ceaea50c6-kube-api-access-hw5p6\") pod \"kuadrant-operator-catalog-dpx8r\" (UID: \"77c86f3f-cd04-4cf2-84fe-a82ceaea50c6\") " pod="kuadrant-system/kuadrant-operator-catalog-dpx8r" Jul 08 17:45:14.761413 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:14.761383 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dpx8r" Jul 08 17:45:14.796216 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:14.796181 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dpx8r"] Jul 08 17:45:14.899024 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:14.898997 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dpx8r"] Jul 08 17:45:14.900808 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:45:14.900778 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c86f3f_cd04_4cf2_84fe_a82ceaea50c6.slice/crio-35c7f299464dd336d2a6e8f31055e8457faf0768ec940eab42567538a5239b10 WatchSource:0}: Error finding container 35c7f299464dd336d2a6e8f31055e8457faf0768ec940eab42567538a5239b10: Status 404 returned error can't find the container with id 35c7f299464dd336d2a6e8f31055e8457faf0768ec940eab42567538a5239b10 Jul 08 17:45:15.019934 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:15.019899 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-94dzc"] Jul 08 17:45:15.024321 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:15.024302 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-94dzc" Jul 08 17:45:15.042061 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:15.042034 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-94dzc"] Jul 08 17:45:15.085020 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:15.084970 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfhh5\" (UniqueName: \"kubernetes.io/projected/4a3e7aea-8da6-4392-a897-bcd5d475a640-kube-api-access-nfhh5\") pod \"kuadrant-operator-catalog-94dzc\" (UID: \"4a3e7aea-8da6-4392-a897-bcd5d475a640\") " pod="kuadrant-system/kuadrant-operator-catalog-94dzc" Jul 08 17:45:15.186032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:15.185993 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfhh5\" (UniqueName: \"kubernetes.io/projected/4a3e7aea-8da6-4392-a897-bcd5d475a640-kube-api-access-nfhh5\") pod \"kuadrant-operator-catalog-94dzc\" (UID: \"4a3e7aea-8da6-4392-a897-bcd5d475a640\") " pod="kuadrant-system/kuadrant-operator-catalog-94dzc" Jul 08 17:45:15.195322 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:15.195288 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfhh5\" (UniqueName: \"kubernetes.io/projected/4a3e7aea-8da6-4392-a897-bcd5d475a640-kube-api-access-nfhh5\") pod \"kuadrant-operator-catalog-94dzc\" (UID: \"4a3e7aea-8da6-4392-a897-bcd5d475a640\") " pod="kuadrant-system/kuadrant-operator-catalog-94dzc" Jul 08 17:45:15.205468 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:15.205435 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:15.304616 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:15.304580 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:15.333421 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:15.333384 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-94dzc" Jul 08 17:45:15.471066 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:15.471040 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-94dzc"] Jul 08 17:45:15.473288 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:45:15.473257 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a3e7aea_8da6_4392_a897_bcd5d475a640.slice/crio-b5c99c840378bcefa157e7835900d28ff96c9b80141608ae3157ea0fb116f54b WatchSource:0}: Error finding container b5c99c840378bcefa157e7835900d28ff96c9b80141608ae3157ea0fb116f54b: Status 404 returned error can't find the container with id b5c99c840378bcefa157e7835900d28ff96c9b80141608ae3157ea0fb116f54b Jul 08 17:45:15.707088 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:15.707054 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dpx8r" event={"ID":"77c86f3f-cd04-4cf2-84fe-a82ceaea50c6","Type":"ContainerStarted","Data":"35c7f299464dd336d2a6e8f31055e8457faf0768ec940eab42567538a5239b10"} Jul 08 17:45:15.708443 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:15.708411 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-94dzc" event={"ID":"4a3e7aea-8da6-4392-a897-bcd5d475a640","Type":"ContainerStarted","Data":"b5c99c840378bcefa157e7835900d28ff96c9b80141608ae3157ea0fb116f54b"} Jul 08 17:45:17.246229 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:17.246138 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:17.346831 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:17.346787 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:17.719785 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:17.719739 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-94dzc" event={"ID":"4a3e7aea-8da6-4392-a897-bcd5d475a640","Type":"ContainerStarted","Data":"cc10d799f15859c222d1832bb4485eddc16dd939b84416ff5f7e94144692d068"} Jul 08 17:45:17.721211 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:17.721172 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dpx8r" event={"ID":"77c86f3f-cd04-4cf2-84fe-a82ceaea50c6","Type":"ContainerStarted","Data":"b125dfac1bd4b018d09e75434e7d91975ded5bf0d150d7452f601ada6019566c"} Jul 08 17:45:17.721353 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:17.721225 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-dpx8r" podUID="77c86f3f-cd04-4cf2-84fe-a82ceaea50c6" containerName="registry-server" containerID="cri-o://b125dfac1bd4b018d09e75434e7d91975ded5bf0d150d7452f601ada6019566c" gracePeriod=2 Jul 08 17:45:17.736744 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:17.736698 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-94dzc" podStartSLOduration=2.230728401 podStartE2EDuration="3.736678044s" totalImagesPullingTime="1.505949643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:45:17.735819894 +0000 UTC m=+393.419917173" watchObservedRunningTime="2026-07-08 17:45:17.736678044 +0000 UTC m=+393.420775322" Jul 08 17:45:17.752693 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:17.752645 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-dpx8r" podStartSLOduration=1.674432082 podStartE2EDuration="3.752629942s" totalImagesPullingTime="2.07819786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:45:17.751479024 +0000 UTC m=+393.435576301" watchObservedRunningTime="2026-07-08 17:45:17.752629942 +0000 UTC m=+393.436727226" Jul 08 17:45:17.965191 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:17.965166 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dpx8r" Jul 08 17:45:18.110542 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:18.110463 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hw5p6\" (UniqueName: \"kubernetes.io/projected/77c86f3f-cd04-4cf2-84fe-a82ceaea50c6-kube-api-access-hw5p6\") pod \"77c86f3f-cd04-4cf2-84fe-a82ceaea50c6\" (UID: \"77c86f3f-cd04-4cf2-84fe-a82ceaea50c6\") " Jul 08 17:45:18.112609 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:18.112577 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c86f3f-cd04-4cf2-84fe-a82ceaea50c6-kube-api-access-hw5p6" (OuterVolumeSpecName: "kube-api-access-hw5p6") pod "77c86f3f-cd04-4cf2-84fe-a82ceaea50c6" (UID: "77c86f3f-cd04-4cf2-84fe-a82ceaea50c6"). InnerVolumeSpecName "kube-api-access-hw5p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:45:18.211504 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:18.211465 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hw5p6\" (UniqueName: \"kubernetes.io/projected/77c86f3f-cd04-4cf2-84fe-a82ceaea50c6-kube-api-access-hw5p6\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:45:18.726599 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:18.726564 2584 generic.go:356] "Generic (PLEG): container finished" podID="77c86f3f-cd04-4cf2-84fe-a82ceaea50c6" containerID="b125dfac1bd4b018d09e75434e7d91975ded5bf0d150d7452f601ada6019566c" exitCode=0 Jul 08 17:45:18.727001 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:18.726606 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-dpx8r" Jul 08 17:45:18.727001 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:18.726645 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dpx8r" event={"ID":"77c86f3f-cd04-4cf2-84fe-a82ceaea50c6","Type":"ContainerDied","Data":"b125dfac1bd4b018d09e75434e7d91975ded5bf0d150d7452f601ada6019566c"} Jul 08 17:45:18.727001 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:18.726698 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-dpx8r" event={"ID":"77c86f3f-cd04-4cf2-84fe-a82ceaea50c6","Type":"ContainerDied","Data":"35c7f299464dd336d2a6e8f31055e8457faf0768ec940eab42567538a5239b10"} Jul 08 17:45:18.727001 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:18.726724 2584 scope.go:117] "RemoveContainer" containerID="b125dfac1bd4b018d09e75434e7d91975ded5bf0d150d7452f601ada6019566c" Jul 08 17:45:18.736377 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:18.736358 2584 scope.go:117] "RemoveContainer" containerID="b125dfac1bd4b018d09e75434e7d91975ded5bf0d150d7452f601ada6019566c" Jul 08 17:45:18.736616 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:45:18.736591 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b125dfac1bd4b018d09e75434e7d91975ded5bf0d150d7452f601ada6019566c\": container with ID starting with b125dfac1bd4b018d09e75434e7d91975ded5bf0d150d7452f601ada6019566c not found: ID does not exist" containerID="b125dfac1bd4b018d09e75434e7d91975ded5bf0d150d7452f601ada6019566c" Jul 08 17:45:18.736711 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:18.736621 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b125dfac1bd4b018d09e75434e7d91975ded5bf0d150d7452f601ada6019566c"} err="failed to get container status \"b125dfac1bd4b018d09e75434e7d91975ded5bf0d150d7452f601ada6019566c\": rpc error: code = NotFound desc = could not find container \"b125dfac1bd4b018d09e75434e7d91975ded5bf0d150d7452f601ada6019566c\": container with ID starting with b125dfac1bd4b018d09e75434e7d91975ded5bf0d150d7452f601ada6019566c not found: ID does not exist" Jul 08 17:45:18.752632 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:18.752611 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dpx8r"] Jul 08 17:45:18.755090 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:18.755067 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-dpx8r"] Jul 08 17:45:18.956259 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:18.956219 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77c86f3f-cd04-4cf2-84fe-a82ceaea50c6" path="/var/lib/kubelet/pods/77c86f3f-cd04-4cf2-84fe-a82ceaea50c6/volumes" Jul 08 17:45:25.333897 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:25.333860 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-94dzc" Jul 08 17:45:25.334414 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:25.333925 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-94dzc" Jul 08 17:45:25.356455 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:25.356431 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-94dzc" Jul 08 17:45:25.783014 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:25.782986 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-94dzc" Jul 08 17:45:29.249470 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.249436 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25"] Jul 08 17:45:29.249885 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.249785 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77c86f3f-cd04-4cf2-84fe-a82ceaea50c6" containerName="registry-server" Jul 08 17:45:29.249885 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.249799 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="77c86f3f-cd04-4cf2-84fe-a82ceaea50c6" containerName="registry-server" Jul 08 17:45:29.249885 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.249881 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="77c86f3f-cd04-4cf2-84fe-a82ceaea50c6" containerName="registry-server" Jul 08 17:45:29.254543 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.254522 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25" Jul 08 17:45:29.257077 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.257051 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wrtxr\"" Jul 08 17:45:29.263298 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.263273 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25"] Jul 08 17:45:29.409540 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.409508 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0ab5324-45ef-4724-a254-4bd8d0c11316-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25\" (UID: \"e0ab5324-45ef-4724-a254-4bd8d0c11316\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25" Jul 08 17:45:29.409721 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.409578 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65crp\" (UniqueName: \"kubernetes.io/projected/e0ab5324-45ef-4724-a254-4bd8d0c11316-kube-api-access-65crp\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25\" (UID: \"e0ab5324-45ef-4724-a254-4bd8d0c11316\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25" Jul 08 17:45:29.409721 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.409618 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0ab5324-45ef-4724-a254-4bd8d0c11316-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25\" (UID: \"e0ab5324-45ef-4724-a254-4bd8d0c11316\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25" Jul 08 17:45:29.511102 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.511006 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0ab5324-45ef-4724-a254-4bd8d0c11316-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25\" (UID: \"e0ab5324-45ef-4724-a254-4bd8d0c11316\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25" Jul 08 17:45:29.511102 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.511062 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0ab5324-45ef-4724-a254-4bd8d0c11316-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25\" (UID: \"e0ab5324-45ef-4724-a254-4bd8d0c11316\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25" Jul 08 17:45:29.511102 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.511101 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65crp\" (UniqueName: \"kubernetes.io/projected/e0ab5324-45ef-4724-a254-4bd8d0c11316-kube-api-access-65crp\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25\" (UID: \"e0ab5324-45ef-4724-a254-4bd8d0c11316\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25" Jul 08 17:45:29.511460 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.511438 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0ab5324-45ef-4724-a254-4bd8d0c11316-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25\" (UID: \"e0ab5324-45ef-4724-a254-4bd8d0c11316\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25" Jul 08 17:45:29.511550 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.511529 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0ab5324-45ef-4724-a254-4bd8d0c11316-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25\" (UID: \"e0ab5324-45ef-4724-a254-4bd8d0c11316\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25" Jul 08 17:45:29.520980 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.520953 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65crp\" (UniqueName: \"kubernetes.io/projected/e0ab5324-45ef-4724-a254-4bd8d0c11316-kube-api-access-65crp\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25\" (UID: \"e0ab5324-45ef-4724-a254-4bd8d0c11316\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25" Jul 08 17:45:29.563841 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.563806 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25" Jul 08 17:45:29.705722 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.705680 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25"] Jul 08 17:45:29.710618 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:45:29.710590 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0ab5324_45ef_4724_a254_4bd8d0c11316.slice/crio-88e352a655512b263a7f25eb9d75e090b50448bc28445c83d544673a92b369f1 WatchSource:0}: Error finding container 88e352a655512b263a7f25eb9d75e090b50448bc28445c83d544673a92b369f1: Status 404 returned error can't find the container with id 88e352a655512b263a7f25eb9d75e090b50448bc28445c83d544673a92b369f1 Jul 08 17:45:29.782089 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.782055 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25" event={"ID":"e0ab5324-45ef-4724-a254-4bd8d0c11316","Type":"ContainerStarted","Data":"db5e7f01a4e14142c54bc16ff3ae466e73d9f51e9da2a540d0410505c7c37d8d"} Jul 08 17:45:29.782302 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.782118 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25" event={"ID":"e0ab5324-45ef-4724-a254-4bd8d0c11316","Type":"ContainerStarted","Data":"88e352a655512b263a7f25eb9d75e090b50448bc28445c83d544673a92b369f1"} Jul 08 17:45:29.853090 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.853054 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26"] Jul 08 17:45:29.856706 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.856681 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26" Jul 08 17:45:29.866567 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:29.866536 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26"] Jul 08 17:45:30.015339 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.015303 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/017ec1dd-2a2b-47b4-bfc5-d2aa052fc186-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26\" (UID: \"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26" Jul 08 17:45:30.015339 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.015344 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w26l9\" (UniqueName: \"kubernetes.io/projected/017ec1dd-2a2b-47b4-bfc5-d2aa052fc186-kube-api-access-w26l9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26\" (UID: \"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26" Jul 08 17:45:30.015543 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.015414 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/017ec1dd-2a2b-47b4-bfc5-d2aa052fc186-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26\" (UID: \"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26" Jul 08 17:45:30.116543 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.116503 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/017ec1dd-2a2b-47b4-bfc5-d2aa052fc186-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26\" (UID: \"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26" Jul 08 17:45:30.116543 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.116549 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w26l9\" (UniqueName: \"kubernetes.io/projected/017ec1dd-2a2b-47b4-bfc5-d2aa052fc186-kube-api-access-w26l9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26\" (UID: \"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26" Jul 08 17:45:30.116754 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.116589 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/017ec1dd-2a2b-47b4-bfc5-d2aa052fc186-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26\" (UID: \"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26" Jul 08 17:45:30.116929 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.116912 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/017ec1dd-2a2b-47b4-bfc5-d2aa052fc186-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26\" (UID: \"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26" Jul 08 17:45:30.116963 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.116943 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/017ec1dd-2a2b-47b4-bfc5-d2aa052fc186-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26\" (UID: \"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26" Jul 08 17:45:30.126657 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.126630 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w26l9\" (UniqueName: \"kubernetes.io/projected/017ec1dd-2a2b-47b4-bfc5-d2aa052fc186-kube-api-access-w26l9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26\" (UID: \"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26" Jul 08 17:45:30.176616 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.176578 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26" Jul 08 17:45:30.312171 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.312139 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26"] Jul 08 17:45:30.313539 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:45:30.313503 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod017ec1dd_2a2b_47b4_bfc5_d2aa052fc186.slice/crio-f5711793f1e7c362eac791150a4f0cf681e45eb892194a3fcb5e63de9d3b453e WatchSource:0}: Error finding container f5711793f1e7c362eac791150a4f0cf681e45eb892194a3fcb5e63de9d3b453e: Status 404 returned error can't find the container with id f5711793f1e7c362eac791150a4f0cf681e45eb892194a3fcb5e63de9d3b453e Jul 08 17:45:30.659114 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.659035 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b"] Jul 08 17:45:30.662548 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.662529 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b" Jul 08 17:45:30.672137 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.672113 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b"] Jul 08 17:45:30.788335 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.788295 2584 generic.go:356] "Generic (PLEG): container finished" podID="e0ab5324-45ef-4724-a254-4bd8d0c11316" containerID="db5e7f01a4e14142c54bc16ff3ae466e73d9f51e9da2a540d0410505c7c37d8d" exitCode=0 Jul 08 17:45:30.788517 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.788393 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25" event={"ID":"e0ab5324-45ef-4724-a254-4bd8d0c11316","Type":"ContainerDied","Data":"db5e7f01a4e14142c54bc16ff3ae466e73d9f51e9da2a540d0410505c7c37d8d"} Jul 08 17:45:30.790105 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.790079 2584 generic.go:356] "Generic (PLEG): container finished" podID="017ec1dd-2a2b-47b4-bfc5-d2aa052fc186" containerID="2344743f39a79d8d190ba7f90a0c4fdfbfb0adee1c24b14f166885464ce7e9fc" exitCode=0 Jul 08 17:45:30.790195 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.790124 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26" event={"ID":"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186","Type":"ContainerDied","Data":"2344743f39a79d8d190ba7f90a0c4fdfbfb0adee1c24b14f166885464ce7e9fc"} Jul 08 17:45:30.790195 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.790160 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26" event={"ID":"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186","Type":"ContainerStarted","Data":"f5711793f1e7c362eac791150a4f0cf681e45eb892194a3fcb5e63de9d3b453e"} Jul 08 17:45:30.822947 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.822913 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f73576f-4b44-4102-9663-06af69ca2856-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b\" (UID: \"2f73576f-4b44-4102-9663-06af69ca2856\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b" Jul 08 17:45:30.822947 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.822956 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6tq2\" (UniqueName: \"kubernetes.io/projected/2f73576f-4b44-4102-9663-06af69ca2856-kube-api-access-s6tq2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b\" (UID: \"2f73576f-4b44-4102-9663-06af69ca2856\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b" Jul 08 17:45:30.823119 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.823037 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f73576f-4b44-4102-9663-06af69ca2856-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b\" (UID: \"2f73576f-4b44-4102-9663-06af69ca2856\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b" Jul 08 17:45:30.924282 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.924160 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f73576f-4b44-4102-9663-06af69ca2856-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b\" (UID: \"2f73576f-4b44-4102-9663-06af69ca2856\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b" Jul 08 17:45:30.924282 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.924226 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6tq2\" (UniqueName: \"kubernetes.io/projected/2f73576f-4b44-4102-9663-06af69ca2856-kube-api-access-s6tq2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b\" (UID: \"2f73576f-4b44-4102-9663-06af69ca2856\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b" Jul 08 17:45:30.924527 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.924308 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f73576f-4b44-4102-9663-06af69ca2856-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b\" (UID: \"2f73576f-4b44-4102-9663-06af69ca2856\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b" Jul 08 17:45:30.924616 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.924593 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f73576f-4b44-4102-9663-06af69ca2856-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b\" (UID: \"2f73576f-4b44-4102-9663-06af69ca2856\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b" Jul 08 17:45:30.924675 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.924617 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f73576f-4b44-4102-9663-06af69ca2856-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b\" (UID: \"2f73576f-4b44-4102-9663-06af69ca2856\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b" Jul 08 17:45:30.934849 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.934828 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6tq2\" (UniqueName: \"kubernetes.io/projected/2f73576f-4b44-4102-9663-06af69ca2856-kube-api-access-s6tq2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b\" (UID: \"2f73576f-4b44-4102-9663-06af69ca2856\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b" Jul 08 17:45:30.971733 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:30.971693 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b" Jul 08 17:45:31.111439 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:31.111415 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b"] Jul 08 17:45:31.113055 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:45:31.113025 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f73576f_4b44_4102_9663_06af69ca2856.slice/crio-c133f4eeea04478c1cb00b86016f0bc87634437e82039969502a8f4d406b27e6 WatchSource:0}: Error finding container c133f4eeea04478c1cb00b86016f0bc87634437e82039969502a8f4d406b27e6: Status 404 returned error can't find the container with id c133f4eeea04478c1cb00b86016f0bc87634437e82039969502a8f4d406b27e6 Jul 08 17:45:31.463820 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:31.463737 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7"] Jul 08 17:45:31.467170 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:31.467141 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7" Jul 08 17:45:31.477890 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:31.477859 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7"] Jul 08 17:45:31.630218 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:31.630179 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf6jk\" (UniqueName: \"kubernetes.io/projected/03f69836-4f17-4951-865c-8998a1d10df1-kube-api-access-kf6jk\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7\" (UID: \"03f69836-4f17-4951-865c-8998a1d10df1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7" Jul 08 17:45:31.630218 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:31.630220 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03f69836-4f17-4951-865c-8998a1d10df1-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7\" (UID: \"03f69836-4f17-4951-865c-8998a1d10df1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7" Jul 08 17:45:31.630434 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:31.630305 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03f69836-4f17-4951-865c-8998a1d10df1-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7\" (UID: \"03f69836-4f17-4951-865c-8998a1d10df1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7" Jul 08 17:45:31.730943 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:31.730839 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03f69836-4f17-4951-865c-8998a1d10df1-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7\" (UID: \"03f69836-4f17-4951-865c-8998a1d10df1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7" Jul 08 17:45:31.731081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:31.730943 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf6jk\" (UniqueName: \"kubernetes.io/projected/03f69836-4f17-4951-865c-8998a1d10df1-kube-api-access-kf6jk\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7\" (UID: \"03f69836-4f17-4951-865c-8998a1d10df1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7" Jul 08 17:45:31.731081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:31.730974 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03f69836-4f17-4951-865c-8998a1d10df1-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7\" (UID: \"03f69836-4f17-4951-865c-8998a1d10df1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7" Jul 08 17:45:31.731340 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:31.731316 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03f69836-4f17-4951-865c-8998a1d10df1-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7\" (UID: \"03f69836-4f17-4951-865c-8998a1d10df1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7" Jul 08 17:45:31.731425 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:31.731319 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03f69836-4f17-4951-865c-8998a1d10df1-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7\" (UID: \"03f69836-4f17-4951-865c-8998a1d10df1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7" Jul 08 17:45:31.741049 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:31.741025 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf6jk\" (UniqueName: \"kubernetes.io/projected/03f69836-4f17-4951-865c-8998a1d10df1-kube-api-access-kf6jk\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7\" (UID: \"03f69836-4f17-4951-865c-8998a1d10df1\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7" Jul 08 17:45:31.776035 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:31.776009 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7" Jul 08 17:45:31.796938 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:31.796904 2584 generic.go:356] "Generic (PLEG): container finished" podID="2f73576f-4b44-4102-9663-06af69ca2856" containerID="cddfab5d81c1e9ed44224d34054e59ef243d63eda3e5bcdf22603a1fa59c8ad6" exitCode=0 Jul 08 17:45:31.797062 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:31.796988 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b" event={"ID":"2f73576f-4b44-4102-9663-06af69ca2856","Type":"ContainerDied","Data":"cddfab5d81c1e9ed44224d34054e59ef243d63eda3e5bcdf22603a1fa59c8ad6"} Jul 08 17:45:31.797062 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:31.797054 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b" event={"ID":"2f73576f-4b44-4102-9663-06af69ca2856","Type":"ContainerStarted","Data":"c133f4eeea04478c1cb00b86016f0bc87634437e82039969502a8f4d406b27e6"} Jul 08 17:45:31.926178 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:31.926154 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7"] Jul 08 17:45:31.947082 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:45:31.947048 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03f69836_4f17_4951_865c_8998a1d10df1.slice/crio-c82ab55a41ab6a9b7286f5f64aae547b512f5f4b5205c890eb1651d5abbbe2e7 WatchSource:0}: Error finding container c82ab55a41ab6a9b7286f5f64aae547b512f5f4b5205c890eb1651d5abbbe2e7: Status 404 returned error can't find the container with id c82ab55a41ab6a9b7286f5f64aae547b512f5f4b5205c890eb1651d5abbbe2e7 Jul 08 17:45:32.803804 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:32.803766 2584 generic.go:356] "Generic (PLEG): container finished" podID="03f69836-4f17-4951-865c-8998a1d10df1" containerID="dff47b22a556def453279c648e537319f26f11dd61345f5338df2e532a2d0794" exitCode=0 Jul 08 17:45:32.804222 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:32.803861 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7" event={"ID":"03f69836-4f17-4951-865c-8998a1d10df1","Type":"ContainerDied","Data":"dff47b22a556def453279c648e537319f26f11dd61345f5338df2e532a2d0794"} Jul 08 17:45:32.804222 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:32.803950 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7" event={"ID":"03f69836-4f17-4951-865c-8998a1d10df1","Type":"ContainerStarted","Data":"c82ab55a41ab6a9b7286f5f64aae547b512f5f4b5205c890eb1651d5abbbe2e7"} Jul 08 17:45:32.805669 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:32.805644 2584 generic.go:356] "Generic (PLEG): container finished" podID="2f73576f-4b44-4102-9663-06af69ca2856" containerID="f385cb41c628c8ee36be1bc7eeaaac917c9735b391a8e32909a2bf6ae797d6a1" exitCode=0 Jul 08 17:45:32.805788 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:32.805735 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b" event={"ID":"2f73576f-4b44-4102-9663-06af69ca2856","Type":"ContainerDied","Data":"f385cb41c628c8ee36be1bc7eeaaac917c9735b391a8e32909a2bf6ae797d6a1"} Jul 08 17:45:32.807599 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:32.807573 2584 generic.go:356] "Generic (PLEG): container finished" podID="e0ab5324-45ef-4724-a254-4bd8d0c11316" containerID="2f838da9566d31eed7f1295a185767f49185b74cf7b5f6c2338052f5c5f8fa5e" exitCode=0 Jul 08 17:45:32.807697 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:32.807649 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25" event={"ID":"e0ab5324-45ef-4724-a254-4bd8d0c11316","Type":"ContainerDied","Data":"2f838da9566d31eed7f1295a185767f49185b74cf7b5f6c2338052f5c5f8fa5e"} Jul 08 17:45:32.809527 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:32.809508 2584 generic.go:356] "Generic (PLEG): container finished" podID="017ec1dd-2a2b-47b4-bfc5-d2aa052fc186" containerID="da7cd40ee003914aa3d1038f1878377d0a9d2def541e2b2c493d5ac19399f4fa" exitCode=0 Jul 08 17:45:32.809593 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:32.809565 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26" event={"ID":"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186","Type":"ContainerDied","Data":"da7cd40ee003914aa3d1038f1878377d0a9d2def541e2b2c493d5ac19399f4fa"} Jul 08 17:45:33.816707 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:33.816674 2584 generic.go:356] "Generic (PLEG): container finished" podID="03f69836-4f17-4951-865c-8998a1d10df1" containerID="3327993c982c5b6b0e94a8fa77fbd0c4dad1c54248330831d1633c3ff3134f46" exitCode=0 Jul 08 17:45:33.817139 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:33.816750 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7" event={"ID":"03f69836-4f17-4951-865c-8998a1d10df1","Type":"ContainerDied","Data":"3327993c982c5b6b0e94a8fa77fbd0c4dad1c54248330831d1633c3ff3134f46"} Jul 08 17:45:33.819186 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:33.819161 2584 generic.go:356] "Generic (PLEG): container finished" podID="2f73576f-4b44-4102-9663-06af69ca2856" containerID="19e8083d5d2c20201d838a9a7a03b5611561cc7a45312d7474b311d035f9988a" exitCode=0 Jul 08 17:45:33.819317 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:33.819262 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b" event={"ID":"2f73576f-4b44-4102-9663-06af69ca2856","Type":"ContainerDied","Data":"19e8083d5d2c20201d838a9a7a03b5611561cc7a45312d7474b311d035f9988a"} Jul 08 17:45:33.821111 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:33.821089 2584 generic.go:356] "Generic (PLEG): container finished" podID="e0ab5324-45ef-4724-a254-4bd8d0c11316" containerID="467c4a413ba99e07a209d15f606276c684c4fc6c38638e88f012bdcc44f343a4" exitCode=0 Jul 08 17:45:33.821189 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:33.821111 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25" event={"ID":"e0ab5324-45ef-4724-a254-4bd8d0c11316","Type":"ContainerDied","Data":"467c4a413ba99e07a209d15f606276c684c4fc6c38638e88f012bdcc44f343a4"} Jul 08 17:45:33.823286 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:33.823263 2584 generic.go:356] "Generic (PLEG): container finished" podID="017ec1dd-2a2b-47b4-bfc5-d2aa052fc186" containerID="a2577b6d1ddac67ba813591ee161331d0f522618559dcf55efb693cb848fe1d1" exitCode=0 Jul 08 17:45:33.823405 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:33.823295 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26" event={"ID":"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186","Type":"ContainerDied","Data":"a2577b6d1ddac67ba813591ee161331d0f522618559dcf55efb693cb848fe1d1"} Jul 08 17:45:34.017698 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:34.017661 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:34.161381 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:34.161325 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:34.222270 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:34.222207 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:34.758306 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:34.758264 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:34.831378 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:34.831341 2584 generic.go:356] "Generic (PLEG): container finished" podID="03f69836-4f17-4951-865c-8998a1d10df1" containerID="e1371d790b3a0cca8aa5fd13a248de136fe8ec0b818a63b9c1503a4018b598bb" exitCode=0 Jul 08 17:45:34.831808 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:34.831385 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7" event={"ID":"03f69836-4f17-4951-865c-8998a1d10df1","Type":"ContainerDied","Data":"e1371d790b3a0cca8aa5fd13a248de136fe8ec0b818a63b9c1503a4018b598bb"} Jul 08 17:45:34.988882 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:34.988850 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b" Jul 08 17:45:35.039342 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.039304 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26" Jul 08 17:45:35.044000 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.043976 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25" Jul 08 17:45:35.161478 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.161445 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0ab5324-45ef-4724-a254-4bd8d0c11316-bundle\") pod \"e0ab5324-45ef-4724-a254-4bd8d0c11316\" (UID: \"e0ab5324-45ef-4724-a254-4bd8d0c11316\") " Jul 08 17:45:35.161478 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.161489 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f73576f-4b44-4102-9663-06af69ca2856-bundle\") pod \"2f73576f-4b44-4102-9663-06af69ca2856\" (UID: \"2f73576f-4b44-4102-9663-06af69ca2856\") " Jul 08 17:45:35.161726 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.161525 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/017ec1dd-2a2b-47b4-bfc5-d2aa052fc186-bundle\") pod \"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186\" (UID: \"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186\") " Jul 08 17:45:35.161726 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.161576 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/017ec1dd-2a2b-47b4-bfc5-d2aa052fc186-util\") pod \"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186\" (UID: \"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186\") " Jul 08 17:45:35.161726 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.161608 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6tq2\" (UniqueName: \"kubernetes.io/projected/2f73576f-4b44-4102-9663-06af69ca2856-kube-api-access-s6tq2\") pod \"2f73576f-4b44-4102-9663-06af69ca2856\" (UID: \"2f73576f-4b44-4102-9663-06af69ca2856\") " Jul 08 17:45:35.161726 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.161659 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0ab5324-45ef-4724-a254-4bd8d0c11316-util\") pod \"e0ab5324-45ef-4724-a254-4bd8d0c11316\" (UID: \"e0ab5324-45ef-4724-a254-4bd8d0c11316\") " Jul 08 17:45:35.161726 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.161682 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w26l9\" (UniqueName: \"kubernetes.io/projected/017ec1dd-2a2b-47b4-bfc5-d2aa052fc186-kube-api-access-w26l9\") pod \"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186\" (UID: \"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186\") " Jul 08 17:45:35.161726 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.161710 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65crp\" (UniqueName: \"kubernetes.io/projected/e0ab5324-45ef-4724-a254-4bd8d0c11316-kube-api-access-65crp\") pod \"e0ab5324-45ef-4724-a254-4bd8d0c11316\" (UID: \"e0ab5324-45ef-4724-a254-4bd8d0c11316\") " Jul 08 17:45:35.162367 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.162061 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f73576f-4b44-4102-9663-06af69ca2856-util\") pod \"2f73576f-4b44-4102-9663-06af69ca2856\" (UID: \"2f73576f-4b44-4102-9663-06af69ca2856\") " Jul 08 17:45:35.162367 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.162131 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ab5324-45ef-4724-a254-4bd8d0c11316-bundle" (OuterVolumeSpecName: "bundle") pod "e0ab5324-45ef-4724-a254-4bd8d0c11316" (UID: "e0ab5324-45ef-4724-a254-4bd8d0c11316"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:45:35.162367 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.162314 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017ec1dd-2a2b-47b4-bfc5-d2aa052fc186-bundle" (OuterVolumeSpecName: "bundle") pod "017ec1dd-2a2b-47b4-bfc5-d2aa052fc186" (UID: "017ec1dd-2a2b-47b4-bfc5-d2aa052fc186"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:45:35.162367 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.162328 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f73576f-4b44-4102-9663-06af69ca2856-bundle" (OuterVolumeSpecName: "bundle") pod "2f73576f-4b44-4102-9663-06af69ca2856" (UID: "2f73576f-4b44-4102-9663-06af69ca2856"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:45:35.162842 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.162406 2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/017ec1dd-2a2b-47b4-bfc5-d2aa052fc186-bundle\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:45:35.162842 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.162427 2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0ab5324-45ef-4724-a254-4bd8d0c11316-bundle\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:45:35.162842 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.162439 2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f73576f-4b44-4102-9663-06af69ca2856-bundle\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:45:35.164151 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.164122 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f73576f-4b44-4102-9663-06af69ca2856-kube-api-access-s6tq2" (OuterVolumeSpecName: "kube-api-access-s6tq2") pod "2f73576f-4b44-4102-9663-06af69ca2856" (UID: "2f73576f-4b44-4102-9663-06af69ca2856"). InnerVolumeSpecName "kube-api-access-s6tq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:45:35.164326 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.164277 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0ab5324-45ef-4724-a254-4bd8d0c11316-kube-api-access-65crp" (OuterVolumeSpecName: "kube-api-access-65crp") pod "e0ab5324-45ef-4724-a254-4bd8d0c11316" (UID: "e0ab5324-45ef-4724-a254-4bd8d0c11316"). InnerVolumeSpecName "kube-api-access-65crp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:45:35.164731 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.164704 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017ec1dd-2a2b-47b4-bfc5-d2aa052fc186-kube-api-access-w26l9" (OuterVolumeSpecName: "kube-api-access-w26l9") pod "017ec1dd-2a2b-47b4-bfc5-d2aa052fc186" (UID: "017ec1dd-2a2b-47b4-bfc5-d2aa052fc186"). InnerVolumeSpecName "kube-api-access-w26l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:45:35.168381 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.168353 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/017ec1dd-2a2b-47b4-bfc5-d2aa052fc186-util" (OuterVolumeSpecName: "util") pod "017ec1dd-2a2b-47b4-bfc5-d2aa052fc186" (UID: "017ec1dd-2a2b-47b4-bfc5-d2aa052fc186"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:45:35.170062 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.170029 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f73576f-4b44-4102-9663-06af69ca2856-util" (OuterVolumeSpecName: "util") pod "2f73576f-4b44-4102-9663-06af69ca2856" (UID: "2f73576f-4b44-4102-9663-06af69ca2856"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:45:35.170209 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.170192 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0ab5324-45ef-4724-a254-4bd8d0c11316-util" (OuterVolumeSpecName: "util") pod "e0ab5324-45ef-4724-a254-4bd8d0c11316" (UID: "e0ab5324-45ef-4724-a254-4bd8d0c11316"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:45:35.263149 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.263046 2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f73576f-4b44-4102-9663-06af69ca2856-util\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:45:35.263149 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.263090 2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/017ec1dd-2a2b-47b4-bfc5-d2aa052fc186-util\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:45:35.263149 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.263105 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6tq2\" (UniqueName: \"kubernetes.io/projected/2f73576f-4b44-4102-9663-06af69ca2856-kube-api-access-s6tq2\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:45:35.263149 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.263118 2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0ab5324-45ef-4724-a254-4bd8d0c11316-util\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:45:35.263149 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.263130 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w26l9\" (UniqueName: \"kubernetes.io/projected/017ec1dd-2a2b-47b4-bfc5-d2aa052fc186-kube-api-access-w26l9\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:45:35.263149 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.263143 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65crp\" (UniqueName: \"kubernetes.io/projected/e0ab5324-45ef-4724-a254-4bd8d0c11316-kube-api-access-65crp\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:45:35.838507 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.838484 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b" Jul 08 17:45:35.838906 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.838478 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b" event={"ID":"2f73576f-4b44-4102-9663-06af69ca2856","Type":"ContainerDied","Data":"c133f4eeea04478c1cb00b86016f0bc87634437e82039969502a8f4d406b27e6"} Jul 08 17:45:35.838906 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.838609 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c133f4eeea04478c1cb00b86016f0bc87634437e82039969502a8f4d406b27e6" Jul 08 17:45:35.840463 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.840437 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25" event={"ID":"e0ab5324-45ef-4724-a254-4bd8d0c11316","Type":"ContainerDied","Data":"88e352a655512b263a7f25eb9d75e090b50448bc28445c83d544673a92b369f1"} Jul 08 17:45:35.840589 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.840476 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88e352a655512b263a7f25eb9d75e090b50448bc28445c83d544673a92b369f1" Jul 08 17:45:35.840589 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.840444 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25" Jul 08 17:45:35.842434 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.842404 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26" event={"ID":"017ec1dd-2a2b-47b4-bfc5-d2aa052fc186","Type":"ContainerDied","Data":"f5711793f1e7c362eac791150a4f0cf681e45eb892194a3fcb5e63de9d3b453e"} Jul 08 17:45:35.842548 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.842450 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5711793f1e7c362eac791150a4f0cf681e45eb892194a3fcb5e63de9d3b453e" Jul 08 17:45:35.842643 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.842626 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26" Jul 08 17:45:35.979650 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:35.979619 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7" Jul 08 17:45:36.070512 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:36.070475 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf6jk\" (UniqueName: \"kubernetes.io/projected/03f69836-4f17-4951-865c-8998a1d10df1-kube-api-access-kf6jk\") pod \"03f69836-4f17-4951-865c-8998a1d10df1\" (UID: \"03f69836-4f17-4951-865c-8998a1d10df1\") " Jul 08 17:45:36.070512 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:36.070513 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03f69836-4f17-4951-865c-8998a1d10df1-bundle\") pod \"03f69836-4f17-4951-865c-8998a1d10df1\" (UID: \"03f69836-4f17-4951-865c-8998a1d10df1\") " Jul 08 17:45:36.070722 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:36.070569 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03f69836-4f17-4951-865c-8998a1d10df1-util\") pod \"03f69836-4f17-4951-865c-8998a1d10df1\" (UID: \"03f69836-4f17-4951-865c-8998a1d10df1\") " Jul 08 17:45:36.071084 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:36.071055 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f69836-4f17-4951-865c-8998a1d10df1-bundle" (OuterVolumeSpecName: "bundle") pod "03f69836-4f17-4951-865c-8998a1d10df1" (UID: "03f69836-4f17-4951-865c-8998a1d10df1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:45:36.072709 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:36.072690 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03f69836-4f17-4951-865c-8998a1d10df1-kube-api-access-kf6jk" (OuterVolumeSpecName: "kube-api-access-kf6jk") pod "03f69836-4f17-4951-865c-8998a1d10df1" (UID: "03f69836-4f17-4951-865c-8998a1d10df1"). InnerVolumeSpecName "kube-api-access-kf6jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:45:36.078478 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:36.078446 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03f69836-4f17-4951-865c-8998a1d10df1-util" (OuterVolumeSpecName: "util") pod "03f69836-4f17-4951-865c-8998a1d10df1" (UID: "03f69836-4f17-4951-865c-8998a1d10df1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:45:36.171557 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:36.171505 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf6jk\" (UniqueName: \"kubernetes.io/projected/03f69836-4f17-4951-865c-8998a1d10df1-kube-api-access-kf6jk\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:45:36.171557 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:36.171551 2584 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03f69836-4f17-4951-865c-8998a1d10df1-bundle\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:45:36.171557 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:36.171560 2584 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03f69836-4f17-4951-865c-8998a1d10df1-util\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:45:36.849973 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:36.849944 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7" Jul 08 17:45:36.849973 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:36.849961 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7" event={"ID":"03f69836-4f17-4951-865c-8998a1d10df1","Type":"ContainerDied","Data":"c82ab55a41ab6a9b7286f5f64aae547b512f5f4b5205c890eb1651d5abbbe2e7"} Jul 08 17:45:36.850520 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:36.850015 2584 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c82ab55a41ab6a9b7286f5f64aae547b512f5f4b5205c890eb1651d5abbbe2e7" Jul 08 17:45:42.854023 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:42.853984 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:43.448437 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:43.448398 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:44.823624 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:44.823581 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:44.922545 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:44.922503 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:45.021195 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:45.021156 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:46.929384 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929353 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ddxqk"] Jul 08 17:45:46.929752 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929694 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0ab5324-45ef-4724-a254-4bd8d0c11316" containerName="extract" Jul 08 17:45:46.929752 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929708 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0ab5324-45ef-4724-a254-4bd8d0c11316" containerName="extract" Jul 08 17:45:46.929752 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929722 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="017ec1dd-2a2b-47b4-bfc5-d2aa052fc186" containerName="util" Jul 08 17:45:46.929752 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929728 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="017ec1dd-2a2b-47b4-bfc5-d2aa052fc186" containerName="util" Jul 08 17:45:46.929752 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929737 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0ab5324-45ef-4724-a254-4bd8d0c11316" containerName="util" Jul 08 17:45:46.929752 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929743 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0ab5324-45ef-4724-a254-4bd8d0c11316" containerName="util" Jul 08 17:45:46.929752 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929749 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f69836-4f17-4951-865c-8998a1d10df1" containerName="extract" Jul 08 17:45:46.929752 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929755 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f69836-4f17-4951-865c-8998a1d10df1" containerName="extract" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929762 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f69836-4f17-4951-865c-8998a1d10df1" containerName="util" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929768 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f69836-4f17-4951-865c-8998a1d10df1" containerName="util" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929776 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="017ec1dd-2a2b-47b4-bfc5-d2aa052fc186" containerName="pull" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929783 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="017ec1dd-2a2b-47b4-bfc5-d2aa052fc186" containerName="pull" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929791 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0ab5324-45ef-4724-a254-4bd8d0c11316" containerName="pull" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929799 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0ab5324-45ef-4724-a254-4bd8d0c11316" containerName="pull" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929805 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03f69836-4f17-4951-865c-8998a1d10df1" containerName="pull" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929811 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="03f69836-4f17-4951-865c-8998a1d10df1" containerName="pull" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929821 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f73576f-4b44-4102-9663-06af69ca2856" containerName="pull" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929825 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f73576f-4b44-4102-9663-06af69ca2856" containerName="pull" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929831 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f73576f-4b44-4102-9663-06af69ca2856" containerName="util" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929836 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f73576f-4b44-4102-9663-06af69ca2856" containerName="util" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929842 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f73576f-4b44-4102-9663-06af69ca2856" containerName="extract" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929847 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f73576f-4b44-4102-9663-06af69ca2856" containerName="extract" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929853 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="017ec1dd-2a2b-47b4-bfc5-d2aa052fc186" containerName="extract" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929858 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="017ec1dd-2a2b-47b4-bfc5-d2aa052fc186" containerName="extract" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929928 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="017ec1dd-2a2b-47b4-bfc5-d2aa052fc186" containerName="extract" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929939 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f73576f-4b44-4102-9663-06af69ca2856" containerName="extract" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929948 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="03f69836-4f17-4951-865c-8998a1d10df1" containerName="extract" Jul 08 17:45:46.930032 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.929959 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0ab5324-45ef-4724-a254-4bd8d0c11316" containerName="extract" Jul 08 17:45:46.934108 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.934091 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ddxqk" Jul 08 17:45:46.936399 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.936382 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-9jpx9\"" Jul 08 17:45:46.943908 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.943882 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ddxqk"] Jul 08 17:45:46.969514 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:46.969483 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmtnt\" (UniqueName: \"kubernetes.io/projected/a47d1f9b-ab0a-4af1-b9b6-3b8988203231-kube-api-access-pmtnt\") pod \"authorino-operator-75cc8d5cb7-ddxqk\" (UID: \"a47d1f9b-ab0a-4af1-b9b6-3b8988203231\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ddxqk" Jul 08 17:45:47.070851 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:47.070806 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmtnt\" (UniqueName: \"kubernetes.io/projected/a47d1f9b-ab0a-4af1-b9b6-3b8988203231-kube-api-access-pmtnt\") pod \"authorino-operator-75cc8d5cb7-ddxqk\" (UID: \"a47d1f9b-ab0a-4af1-b9b6-3b8988203231\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ddxqk" Jul 08 17:45:47.085992 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:47.085964 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmtnt\" (UniqueName: \"kubernetes.io/projected/a47d1f9b-ab0a-4af1-b9b6-3b8988203231-kube-api-access-pmtnt\") pod \"authorino-operator-75cc8d5cb7-ddxqk\" (UID: \"a47d1f9b-ab0a-4af1-b9b6-3b8988203231\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-ddxqk" Jul 08 17:45:47.244351 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:47.244256 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ddxqk" Jul 08 17:45:47.386764 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:47.386737 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-ddxqk"] Jul 08 17:45:47.388868 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:45:47.388835 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47d1f9b_ab0a_4af1_b9b6_3b8988203231.slice/crio-193cf8933bd26005cb3dc18e2235fd371762f0b8a1c2cc33a2ee01ec44ecc05a WatchSource:0}: Error finding container 193cf8933bd26005cb3dc18e2235fd371762f0b8a1c2cc33a2ee01ec44ecc05a: Status 404 returned error can't find the container with id 193cf8933bd26005cb3dc18e2235fd371762f0b8a1c2cc33a2ee01ec44ecc05a Jul 08 17:45:47.907067 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:47.907026 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ddxqk" event={"ID":"a47d1f9b-ab0a-4af1-b9b6-3b8988203231","Type":"ContainerStarted","Data":"193cf8933bd26005cb3dc18e2235fd371762f0b8a1c2cc33a2ee01ec44ecc05a"} Jul 08 17:45:48.246712 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:48.246601 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:49.918859 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:49.918771 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ddxqk" event={"ID":"a47d1f9b-ab0a-4af1-b9b6-3b8988203231","Type":"ContainerStarted","Data":"2a83bef84285cecdfd180d868794bd90ab4e3061b3044a287cb8ae8bcbd3857c"} Jul 08 17:45:49.919235 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:49.918914 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ddxqk" Jul 08 17:45:49.938780 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:49.938725 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ddxqk" podStartSLOduration=1.666291049 podStartE2EDuration="3.938709843s" totalImagesPullingTime="2.272418794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:45:49.937481379 +0000 UTC m=+425.621578666" watchObservedRunningTime="2026-07-08 17:45:49.938709843 +0000 UTC m=+425.622807119" Jul 08 17:45:50.205665 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:50.205583 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl"] Jul 08 17:45:50.209060 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:50.209041 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" Jul 08 17:45:50.211381 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:50.211355 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dp8m5\"" Jul 08 17:45:50.221363 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:50.221342 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl"] Jul 08 17:45:50.298423 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:50.298383 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7gbb\" (UniqueName: \"kubernetes.io/projected/cac6c36b-20e4-463f-93bf-694c0aed137f-kube-api-access-z7gbb\") pod \"limitador-operator-controller-manager-859d7b78bd-sq8fl\" (UID: \"cac6c36b-20e4-463f-93bf-694c0aed137f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" Jul 08 17:45:50.399358 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:50.399316 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gbb\" (UniqueName: \"kubernetes.io/projected/cac6c36b-20e4-463f-93bf-694c0aed137f-kube-api-access-z7gbb\") pod \"limitador-operator-controller-manager-859d7b78bd-sq8fl\" (UID: \"cac6c36b-20e4-463f-93bf-694c0aed137f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" Jul 08 17:45:50.414657 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:50.414631 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7gbb\" (UniqueName: \"kubernetes.io/projected/cac6c36b-20e4-463f-93bf-694c0aed137f-kube-api-access-z7gbb\") pod \"limitador-operator-controller-manager-859d7b78bd-sq8fl\" (UID: \"cac6c36b-20e4-463f-93bf-694c0aed137f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" Jul 08 17:45:50.519305 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:50.519203 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" Jul 08 17:45:50.580742 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:50.580707 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:50.654116 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:50.654089 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl"] Jul 08 17:45:50.655893 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:45:50.655859 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac6c36b_20e4_463f_93bf_694c0aed137f.slice/crio-6d2d7e6c13b9949fd0bacd9bcd0364d0b5fddf8582ca1c2311eb2927cfd0f959 WatchSource:0}: Error finding container 6d2d7e6c13b9949fd0bacd9bcd0364d0b5fddf8582ca1c2311eb2927cfd0f959: Status 404 returned error can't find the container with id 6d2d7e6c13b9949fd0bacd9bcd0364d0b5fddf8582ca1c2311eb2927cfd0f959 Jul 08 17:45:50.660460 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:50.660413 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:50.925189 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:50.925154 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" event={"ID":"cac6c36b-20e4-463f-93bf-694c0aed137f","Type":"ContainerStarted","Data":"6d2d7e6c13b9949fd0bacd9bcd0364d0b5fddf8582ca1c2311eb2927cfd0f959"} Jul 08 17:45:51.248150 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:51.248060 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:52.104436 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:52.104400 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:52.936719 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:52.936687 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" event={"ID":"cac6c36b-20e4-463f-93bf-694c0aed137f","Type":"ContainerStarted","Data":"78fb1eb44e00a7f81924e11708031be18a571ac9fc728960da099f3059e291b0"} Jul 08 17:45:52.936919 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:52.936752 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" Jul 08 17:45:52.954713 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:52.954650 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" podStartSLOduration=1.020802636 podStartE2EDuration="2.954629105s" totalImagesPullingTime="1.933826469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:45:52.952623598 +0000 UTC m=+428.636720888" watchObservedRunningTime="2026-07-08 17:45:52.954629105 +0000 UTC m=+428.638726383" Jul 08 17:45:53.464266 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:53.464213 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bcbfd"] Jul 08 17:45:53.467706 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:53.467686 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bcbfd" Jul 08 17:45:53.469950 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:53.469928 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-wlxjc\"" Jul 08 17:45:53.469950 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:53.469943 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 08 17:45:53.477878 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:53.477852 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bcbfd"] Jul 08 17:45:53.525901 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:53.525870 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf245\" (UniqueName: \"kubernetes.io/projected/efa486df-ee78-4a9e-994a-44722b368f47-kube-api-access-sf245\") pod \"dns-operator-controller-manager-864dd78c4c-bcbfd\" (UID: \"efa486df-ee78-4a9e-994a-44722b368f47\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bcbfd" Jul 08 17:45:53.626394 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:53.626360 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf245\" (UniqueName: \"kubernetes.io/projected/efa486df-ee78-4a9e-994a-44722b368f47-kube-api-access-sf245\") pod \"dns-operator-controller-manager-864dd78c4c-bcbfd\" (UID: \"efa486df-ee78-4a9e-994a-44722b368f47\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bcbfd" Jul 08 17:45:53.639609 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:53.639585 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf245\" (UniqueName: \"kubernetes.io/projected/efa486df-ee78-4a9e-994a-44722b368f47-kube-api-access-sf245\") pod \"dns-operator-controller-manager-864dd78c4c-bcbfd\" (UID: \"efa486df-ee78-4a9e-994a-44722b368f47\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bcbfd" Jul 08 17:45:53.779456 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:53.779364 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bcbfd" Jul 08 17:45:53.914903 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:53.914877 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bcbfd"] Jul 08 17:45:53.917114 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:45:53.917085 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa486df_ee78_4a9e_994a_44722b368f47.slice/crio-90efe3a9c4b183d6222f03c30ea6be14c6e271f46ee2ca0b627beb34428d2dc4 WatchSource:0}: Error finding container 90efe3a9c4b183d6222f03c30ea6be14c6e271f46ee2ca0b627beb34428d2dc4: Status 404 returned error can't find the container with id 90efe3a9c4b183d6222f03c30ea6be14c6e271f46ee2ca0b627beb34428d2dc4 Jul 08 17:45:53.943609 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:53.943578 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bcbfd" event={"ID":"efa486df-ee78-4a9e-994a-44722b368f47","Type":"ContainerStarted","Data":"90efe3a9c4b183d6222f03c30ea6be14c6e271f46ee2ca0b627beb34428d2dc4"} Jul 08 17:45:54.258111 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:54.258073 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:54.692873 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:54.692837 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:55.957621 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:55.957531 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bcbfd" event={"ID":"efa486df-ee78-4a9e-994a-44722b368f47","Type":"ContainerStarted","Data":"89156ca3e0880b6f34485051bce71531e553a8239692ac54b1d1c07e1ab6d36c"} Jul 08 17:45:55.957621 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:55.957607 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bcbfd" Jul 08 17:45:55.981210 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:55.981073 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bcbfd" podStartSLOduration=1.229880381 podStartE2EDuration="2.981056579s" totalImagesPullingTime="1.751176198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:45:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:45:55.980627034 +0000 UTC m=+431.664724311" watchObservedRunningTime="2026-07-08 17:45:55.981056579 +0000 UTC m=+431.665153855" Jul 08 17:45:57.892384 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:57.892348 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:58.403060 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:58.403022 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj"] Jul 08 17:45:58.406608 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:58.406591 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" Jul 08 17:45:58.409367 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:58.409343 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kqmbf\"" Jul 08 17:45:58.419744 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:58.419718 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj"] Jul 08 17:45:58.464872 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:58.464835 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzjrs\" (UniqueName: \"kubernetes.io/projected/40edb0c7-4add-45b6-8d93-f05753b8a2e8-kube-api-access-jzjrs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hfsqj\" (UID: \"40edb0c7-4add-45b6-8d93-f05753b8a2e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" Jul 08 17:45:58.465031 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:58.464882 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40edb0c7-4add-45b6-8d93-f05753b8a2e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hfsqj\" (UID: \"40edb0c7-4add-45b6-8d93-f05753b8a2e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" Jul 08 17:45:58.565888 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:58.565849 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzjrs\" (UniqueName: \"kubernetes.io/projected/40edb0c7-4add-45b6-8d93-f05753b8a2e8-kube-api-access-jzjrs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hfsqj\" (UID: \"40edb0c7-4add-45b6-8d93-f05753b8a2e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" Jul 08 17:45:58.565888 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:58.565892 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40edb0c7-4add-45b6-8d93-f05753b8a2e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hfsqj\" (UID: \"40edb0c7-4add-45b6-8d93-f05753b8a2e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" Jul 08 17:45:58.566279 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:58.566261 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40edb0c7-4add-45b6-8d93-f05753b8a2e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hfsqj\" (UID: \"40edb0c7-4add-45b6-8d93-f05753b8a2e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" Jul 08 17:45:58.575193 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:58.575162 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzjrs\" (UniqueName: \"kubernetes.io/projected/40edb0c7-4add-45b6-8d93-f05753b8a2e8-kube-api-access-jzjrs\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hfsqj\" (UID: \"40edb0c7-4add-45b6-8d93-f05753b8a2e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" Jul 08 17:45:58.716699 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:58.716606 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" Jul 08 17:45:58.857142 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:58.857112 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj"] Jul 08 17:45:58.858341 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:45:58.858314 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40edb0c7_4add_45b6_8d93_f05753b8a2e8.slice/crio-56f9272f59660ee57192dd27ec6ba1e8e4654c92e7792ae68979dbfc8346ce2d WatchSource:0}: Error finding container 56f9272f59660ee57192dd27ec6ba1e8e4654c92e7792ae68979dbfc8346ce2d: Status 404 returned error can't find the container with id 56f9272f59660ee57192dd27ec6ba1e8e4654c92e7792ae68979dbfc8346ce2d Jul 08 17:45:58.973613 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:58.973529 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" event={"ID":"40edb0c7-4add-45b6-8d93-f05753b8a2e8","Type":"ContainerStarted","Data":"56f9272f59660ee57192dd27ec6ba1e8e4654c92e7792ae68979dbfc8346ce2d"} Jul 08 17:45:59.869473 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:59.869425 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:45:59.967232 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:45:59.967189 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:00.265799 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:00.265709 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:00.365852 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:00.365800 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:00.927928 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:00.927900 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-ddxqk" Jul 08 17:46:03.947012 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:03.946977 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" Jul 08 17:46:04.078178 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:04.078137 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:04.174640 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:04.174602 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:05.009469 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:05.009427 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" event={"ID":"40edb0c7-4add-45b6-8d93-f05753b8a2e8","Type":"ContainerStarted","Data":"9b84fc6056efc21477c57523cb5aa8d0f7278b6f72846fb4264aecd15160a993"} Jul 08 17:46:05.009947 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:05.009516 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" Jul 08 17:46:05.038077 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:05.038017 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" podStartSLOduration=1.106351914 podStartE2EDuration="7.037997905s" totalImagesPullingTime="5.931645991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:46:05.034426152 +0000 UTC m=+440.718523429" watchObservedRunningTime="2026-07-08 17:46:05.037997905 +0000 UTC m=+440.722095181" Jul 08 17:46:06.964701 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:06.964665 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bcbfd" Jul 08 17:46:07.072278 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:07.072228 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:07.170081 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:07.170047 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:07.766487 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:07.766450 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9js7v"] Jul 08 17:46:07.768657 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:07.768624 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:07.770339 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:07.770318 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9js7v" Jul 08 17:46:07.772685 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:07.772664 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 08 17:46:07.772813 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:07.772794 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wrtxr\"" Jul 08 17:46:07.773326 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:07.773310 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 08 17:46:07.780711 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:07.780689 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9js7v"] Jul 08 17:46:07.848901 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:07.848873 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2cab7921-a494-44d1-96fe-9f1bf1af5b32-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9js7v\" (UID: \"2cab7921-a494-44d1-96fe-9f1bf1af5b32\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9js7v" Jul 08 17:46:07.849071 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:07.848939 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28wgv\" (UniqueName: \"kubernetes.io/projected/2cab7921-a494-44d1-96fe-9f1bf1af5b32-kube-api-access-28wgv\") pod \"kuadrant-console-plugin-66b5774985-9js7v\" (UID: \"2cab7921-a494-44d1-96fe-9f1bf1af5b32\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9js7v" Jul 08 17:46:07.849071 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:07.848963 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2cab7921-a494-44d1-96fe-9f1bf1af5b32-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9js7v\" (UID: \"2cab7921-a494-44d1-96fe-9f1bf1af5b32\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9js7v" Jul 08 17:46:07.950054 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:07.950021 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2cab7921-a494-44d1-96fe-9f1bf1af5b32-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9js7v\" (UID: \"2cab7921-a494-44d1-96fe-9f1bf1af5b32\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9js7v" Jul 08 17:46:07.950228 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:07.950122 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28wgv\" (UniqueName: \"kubernetes.io/projected/2cab7921-a494-44d1-96fe-9f1bf1af5b32-kube-api-access-28wgv\") pod \"kuadrant-console-plugin-66b5774985-9js7v\" (UID: \"2cab7921-a494-44d1-96fe-9f1bf1af5b32\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9js7v" Jul 08 17:46:07.950228 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:07.950157 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2cab7921-a494-44d1-96fe-9f1bf1af5b32-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9js7v\" (UID: \"2cab7921-a494-44d1-96fe-9f1bf1af5b32\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9js7v" Jul 08 17:46:07.950876 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:07.950854 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2cab7921-a494-44d1-96fe-9f1bf1af5b32-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9js7v\" (UID: \"2cab7921-a494-44d1-96fe-9f1bf1af5b32\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9js7v" Jul 08 17:46:07.952649 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:07.952626 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2cab7921-a494-44d1-96fe-9f1bf1af5b32-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9js7v\" (UID: \"2cab7921-a494-44d1-96fe-9f1bf1af5b32\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9js7v" Jul 08 17:46:07.963167 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:07.963139 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28wgv\" (UniqueName: \"kubernetes.io/projected/2cab7921-a494-44d1-96fe-9f1bf1af5b32-kube-api-access-28wgv\") pod \"kuadrant-console-plugin-66b5774985-9js7v\" (UID: \"2cab7921-a494-44d1-96fe-9f1bf1af5b32\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9js7v" Jul 08 17:46:07.986078 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:07.986042 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:08.080545 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:08.080458 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9js7v" Jul 08 17:46:08.214929 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:08.214903 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9js7v"] Jul 08 17:46:08.216406 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:46:08.216376 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cab7921_a494_44d1_96fe_9f1bf1af5b32.slice/crio-d448e3851819040ca441931903168e564b6d236d90c28955bb4bae0d0a5d9006 WatchSource:0}: Error finding container d448e3851819040ca441931903168e564b6d236d90c28955bb4bae0d0a5d9006: Status 404 returned error can't find the container with id d448e3851819040ca441931903168e564b6d236d90c28955bb4bae0d0a5d9006 Jul 08 17:46:09.034577 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:09.034537 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9js7v" event={"ID":"2cab7921-a494-44d1-96fe-9f1bf1af5b32","Type":"ContainerStarted","Data":"d448e3851819040ca441931903168e564b6d236d90c28955bb4bae0d0a5d9006"} Jul 08 17:46:12.334564 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:12.334520 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:12.436995 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:12.436950 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:16.017682 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:16.017651 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" Jul 08 17:46:17.743671 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.743631 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj"] Jul 08 17:46:17.744169 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.743874 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" podUID="40edb0c7-4add-45b6-8d93-f05753b8a2e8" containerName="manager" containerID="cri-o://9b84fc6056efc21477c57523cb5aa8d0f7278b6f72846fb4264aecd15160a993" gracePeriod=2 Jul 08 17:46:17.750960 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.750921 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj"] Jul 08 17:46:17.764941 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.764507 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl"] Jul 08 17:46:17.764941 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.764745 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" podUID="cac6c36b-20e4-463f-93bf-694c0aed137f" containerName="manager" containerID="cri-o://78fb1eb44e00a7f81924e11708031be18a571ac9fc728960da099f3059e291b0" gracePeriod=2 Jul 08 17:46:17.768220 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.768190 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg"] Jul 08 17:46:17.768843 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.768821 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40edb0c7-4add-45b6-8d93-f05753b8a2e8" containerName="manager" Jul 08 17:46:17.768954 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.768846 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="40edb0c7-4add-45b6-8d93-f05753b8a2e8" containerName="manager" Jul 08 17:46:17.769025 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.768956 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="40edb0c7-4add-45b6-8d93-f05753b8a2e8" containerName="manager" Jul 08 17:46:17.774365 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.774080 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg" Jul 08 17:46:17.774623 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:46:17.774574 2584 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"cac6c36b-20e4-463f-93bf-694c0aed137f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-sq8fl\": pods \"limitador-operator-controller-manager-859d7b78bd-sq8fl\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" Jul 08 17:46:17.775433 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.775386 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl"] Jul 08 17:46:17.776425 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:46:17.776378 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-hfsqj\" is forbidden: User \"system:node:ip-10-0-129-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-226.ec2.internal' and this object" podUID="40edb0c7-4add-45b6-8d93-f05753b8a2e8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" Jul 08 17:46:17.778337 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:46:17.778304 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sq8fl\" is forbidden: User \"system:node:ip-10-0-129-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-226.ec2.internal' and this object" podUID="cac6c36b-20e4-463f-93bf-694c0aed137f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" Jul 08 17:46:17.785585 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.785533 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg"] Jul 08 17:46:17.788989 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.788969 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rd5p5"] Jul 08 17:46:17.789439 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.789418 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cac6c36b-20e4-463f-93bf-694c0aed137f" containerName="manager" Jul 08 17:46:17.789439 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.789442 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="cac6c36b-20e4-463f-93bf-694c0aed137f" containerName="manager" Jul 08 17:46:17.789596 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.789535 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="cac6c36b-20e4-463f-93bf-694c0aed137f" containerName="manager" Jul 08 17:46:17.792982 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.792963 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rd5p5" Jul 08 17:46:17.807431 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.807404 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rd5p5"] Jul 08 17:46:17.817440 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:46:17.817399 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-hfsqj\" is forbidden: User \"system:node:ip-10-0-129-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-226.ec2.internal' and this object" podUID="40edb0c7-4add-45b6-8d93-f05753b8a2e8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" Jul 08 17:46:17.821375 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:46:17.821341 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sq8fl\" is forbidden: User \"system:node:ip-10-0-129-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-226.ec2.internal' and this object" podUID="cac6c36b-20e4-463f-93bf-694c0aed137f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" Jul 08 17:46:17.823397 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:46:17.823365 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-hfsqj\" is forbidden: User \"system:node:ip-10-0-129-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-226.ec2.internal' and this object" podUID="40edb0c7-4add-45b6-8d93-f05753b8a2e8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" Jul 08 17:46:17.844650 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.844618 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzsp5\" (UniqueName: \"kubernetes.io/projected/21d3761b-7a6a-413f-92d4-13ac0e6e042a-kube-api-access-hzsp5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-27stg\" (UID: \"21d3761b-7a6a-413f-92d4-13ac0e6e042a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg" Jul 08 17:46:17.844798 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.844681 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/21d3761b-7a6a-413f-92d4-13ac0e6e042a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-27stg\" (UID: \"21d3761b-7a6a-413f-92d4-13ac0e6e042a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg" Jul 08 17:46:17.945838 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.945797 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzsp5\" (UniqueName: \"kubernetes.io/projected/21d3761b-7a6a-413f-92d4-13ac0e6e042a-kube-api-access-hzsp5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-27stg\" (UID: \"21d3761b-7a6a-413f-92d4-13ac0e6e042a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg" Jul 08 17:46:17.946033 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.945872 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/21d3761b-7a6a-413f-92d4-13ac0e6e042a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-27stg\" (UID: \"21d3761b-7a6a-413f-92d4-13ac0e6e042a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg" Jul 08 17:46:17.946033 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.945938 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff8qq\" (UniqueName: \"kubernetes.io/projected/5f2e7b50-7ca5-43d2-8c6b-44c13a9eb716-kube-api-access-ff8qq\") pod \"limitador-operator-controller-manager-859d7b78bd-rd5p5\" (UID: \"5f2e7b50-7ca5-43d2-8c6b-44c13a9eb716\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rd5p5" Jul 08 17:46:17.946356 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.946331 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/21d3761b-7a6a-413f-92d4-13ac0e6e042a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-27stg\" (UID: \"21d3761b-7a6a-413f-92d4-13ac0e6e042a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg" Jul 08 17:46:17.955832 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:17.955800 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzsp5\" (UniqueName: \"kubernetes.io/projected/21d3761b-7a6a-413f-92d4-13ac0e6e042a-kube-api-access-hzsp5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-27stg\" (UID: \"21d3761b-7a6a-413f-92d4-13ac0e6e042a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg" Jul 08 17:46:18.046776 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:18.046697 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff8qq\" (UniqueName: \"kubernetes.io/projected/5f2e7b50-7ca5-43d2-8c6b-44c13a9eb716-kube-api-access-ff8qq\") pod \"limitador-operator-controller-manager-859d7b78bd-rd5p5\" (UID: \"5f2e7b50-7ca5-43d2-8c6b-44c13a9eb716\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rd5p5" Jul 08 17:46:18.056968 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:18.056937 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff8qq\" (UniqueName: \"kubernetes.io/projected/5f2e7b50-7ca5-43d2-8c6b-44c13a9eb716-kube-api-access-ff8qq\") pod \"limitador-operator-controller-manager-859d7b78bd-rd5p5\" (UID: \"5f2e7b50-7ca5-43d2-8c6b-44c13a9eb716\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rd5p5" Jul 08 17:46:18.157031 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:18.156984 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg" Jul 08 17:46:18.163160 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:18.163134 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rd5p5" Jul 08 17:46:19.275188 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:19.275132 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:19.373715 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:19.373675 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:19.868024 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:19.867988 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:19.971785 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:19.971738 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:23.257473 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:23.257437 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:23.357881 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:23.357840 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:25.672927 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:25.672888 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:25.750292 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:25.750234 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:29.775820 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:29.775757 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:30.979899 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:30.979847 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:31.078798 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:31.078762 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:31.809843 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:31.809812 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg"] Jul 08 17:46:31.813680 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:31.813658 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" Jul 08 17:46:31.816174 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:46:31.816142 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sq8fl\" is forbidden: User \"system:node:ip-10-0-129-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-226.ec2.internal' and this object" podUID="cac6c36b-20e4-463f-93bf-694c0aed137f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" Jul 08 17:46:31.817444 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:46:31.817418 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d3761b_7a6a_413f_92d4_13ac0e6e042a.slice/crio-16985b3e9dc927d88e82ce1bfcb225b0f48e37514a790f216c185b267f51be50 WatchSource:0}: Error finding container 16985b3e9dc927d88e82ce1bfcb225b0f48e37514a790f216c185b267f51be50: Status 404 returned error can't find the container with id 16985b3e9dc927d88e82ce1bfcb225b0f48e37514a790f216c185b267f51be50 Jul 08 17:46:31.824891 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:31.824871 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" Jul 08 17:46:31.827035 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:46:31.827006 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sq8fl\" is forbidden: User \"system:node:ip-10-0-129-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-226.ec2.internal' and this object" podUID="cac6c36b-20e4-463f-93bf-694c0aed137f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" Jul 08 17:46:31.828720 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:46:31.828696 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-hfsqj\" is forbidden: User \"system:node:ip-10-0-129-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-226.ec2.internal' and this object" podUID="40edb0c7-4add-45b6-8d93-f05753b8a2e8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" Jul 08 17:46:31.840186 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:31.840158 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rd5p5"] Jul 08 17:46:31.841823 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:46:31.841798 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2e7b50_7ca5_43d2_8c6b_44c13a9eb716.slice/crio-e6574ce0cff09a3d434c7e27bb5f2d9cb3e2909f0de161e2baf70aba14fbed11 WatchSource:0}: Error finding container e6574ce0cff09a3d434c7e27bb5f2d9cb3e2909f0de161e2baf70aba14fbed11: Status 404 returned error can't find the container with id e6574ce0cff09a3d434c7e27bb5f2d9cb3e2909f0de161e2baf70aba14fbed11 Jul 08 17:46:31.982894 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:31.982856 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7gbb\" (UniqueName: \"kubernetes.io/projected/cac6c36b-20e4-463f-93bf-694c0aed137f-kube-api-access-z7gbb\") pod \"cac6c36b-20e4-463f-93bf-694c0aed137f\" (UID: \"cac6c36b-20e4-463f-93bf-694c0aed137f\") " Jul 08 17:46:31.983308 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:31.982926 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzjrs\" (UniqueName: \"kubernetes.io/projected/40edb0c7-4add-45b6-8d93-f05753b8a2e8-kube-api-access-jzjrs\") pod \"40edb0c7-4add-45b6-8d93-f05753b8a2e8\" (UID: \"40edb0c7-4add-45b6-8d93-f05753b8a2e8\") " Jul 08 17:46:31.983308 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:31.982950 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40edb0c7-4add-45b6-8d93-f05753b8a2e8-extensions-socket-volume\") pod \"40edb0c7-4add-45b6-8d93-f05753b8a2e8\" (UID: \"40edb0c7-4add-45b6-8d93-f05753b8a2e8\") " Jul 08 17:46:31.983513 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:31.983490 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40edb0c7-4add-45b6-8d93-f05753b8a2e8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "40edb0c7-4add-45b6-8d93-f05753b8a2e8" (UID: "40edb0c7-4add-45b6-8d93-f05753b8a2e8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:46:31.984988 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:31.984963 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40edb0c7-4add-45b6-8d93-f05753b8a2e8-kube-api-access-jzjrs" (OuterVolumeSpecName: "kube-api-access-jzjrs") pod "40edb0c7-4add-45b6-8d93-f05753b8a2e8" (UID: "40edb0c7-4add-45b6-8d93-f05753b8a2e8"). InnerVolumeSpecName "kube-api-access-jzjrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:46:31.985055 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:31.984986 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac6c36b-20e4-463f-93bf-694c0aed137f-kube-api-access-z7gbb" (OuterVolumeSpecName: "kube-api-access-z7gbb") pod "cac6c36b-20e4-463f-93bf-694c0aed137f" (UID: "cac6c36b-20e4-463f-93bf-694c0aed137f"). InnerVolumeSpecName "kube-api-access-z7gbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:46:32.084511 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.084473 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzjrs\" (UniqueName: \"kubernetes.io/projected/40edb0c7-4add-45b6-8d93-f05753b8a2e8-kube-api-access-jzjrs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:46:32.084511 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.084503 2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40edb0c7-4add-45b6-8d93-f05753b8a2e8-extensions-socket-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:46:32.084511 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.084514 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7gbb\" (UniqueName: \"kubernetes.io/projected/cac6c36b-20e4-463f-93bf-694c0aed137f-kube-api-access-z7gbb\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:46:32.172398 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.172305 2584 generic.go:356] "Generic (PLEG): container finished" podID="cac6c36b-20e4-463f-93bf-694c0aed137f" containerID="78fb1eb44e00a7f81924e11708031be18a571ac9fc728960da099f3059e291b0" exitCode=0 Jul 08 17:46:32.172398 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.172353 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" Jul 08 17:46:32.172398 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.172396 2584 scope.go:117] "RemoveContainer" containerID="78fb1eb44e00a7f81924e11708031be18a571ac9fc728960da099f3059e291b0" Jul 08 17:46:32.180000 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:46:32.174674 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sq8fl\" is forbidden: User \"system:node:ip-10-0-129-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-226.ec2.internal' and this object" podUID="cac6c36b-20e4-463f-93bf-694c0aed137f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" Jul 08 17:46:32.180000 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:46:32.176343 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-hfsqj\" is forbidden: User \"system:node:ip-10-0-129-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-226.ec2.internal' and this object" podUID="40edb0c7-4add-45b6-8d93-f05753b8a2e8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" Jul 08 17:46:32.180730 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.180696 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9js7v" event={"ID":"2cab7921-a494-44d1-96fe-9f1bf1af5b32","Type":"ContainerStarted","Data":"3f924de13e556df973c513a4b5b29d2c893890c032579bb4ba7c7cf78ad5e43c"} Jul 08 17:46:32.182810 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.182781 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg" event={"ID":"21d3761b-7a6a-413f-92d4-13ac0e6e042a","Type":"ContainerStarted","Data":"8a97e2c63c806bf5825c7ec33c3437673e6a64846689edd20845c56786bd66eb"} Jul 08 17:46:32.182925 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.182868 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg" event={"ID":"21d3761b-7a6a-413f-92d4-13ac0e6e042a","Type":"ContainerStarted","Data":"16985b3e9dc927d88e82ce1bfcb225b0f48e37514a790f216c185b267f51be50"} Jul 08 17:46:32.182925 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.182899 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg" Jul 08 17:46:32.184301 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.184277 2584 generic.go:356] "Generic (PLEG): container finished" podID="40edb0c7-4add-45b6-8d93-f05753b8a2e8" containerID="9b84fc6056efc21477c57523cb5aa8d0f7278b6f72846fb4264aecd15160a993" exitCode=0 Jul 08 17:46:32.184427 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.184380 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" Jul 08 17:46:32.186518 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.186493 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rd5p5" event={"ID":"5f2e7b50-7ca5-43d2-8c6b-44c13a9eb716","Type":"ContainerStarted","Data":"48293f3ab35dda36fec523e6ec5021a73f0b3377ff1c54de7eb3004f82059715"} Jul 08 17:46:32.186618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.186540 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rd5p5" event={"ID":"5f2e7b50-7ca5-43d2-8c6b-44c13a9eb716","Type":"ContainerStarted","Data":"e6574ce0cff09a3d434c7e27bb5f2d9cb3e2909f0de161e2baf70aba14fbed11"} Jul 08 17:46:32.186618 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.186592 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rd5p5" Jul 08 17:46:32.189023 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.189006 2584 scope.go:117] "RemoveContainer" containerID="78fb1eb44e00a7f81924e11708031be18a571ac9fc728960da099f3059e291b0" Jul 08 17:46:32.189292 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:46:32.189271 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78fb1eb44e00a7f81924e11708031be18a571ac9fc728960da099f3059e291b0\": container with ID starting with 78fb1eb44e00a7f81924e11708031be18a571ac9fc728960da099f3059e291b0 not found: ID does not exist" containerID="78fb1eb44e00a7f81924e11708031be18a571ac9fc728960da099f3059e291b0" Jul 08 17:46:32.189376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.189299 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78fb1eb44e00a7f81924e11708031be18a571ac9fc728960da099f3059e291b0"} err="failed to get container status \"78fb1eb44e00a7f81924e11708031be18a571ac9fc728960da099f3059e291b0\": rpc error: code = NotFound desc = could not find container \"78fb1eb44e00a7f81924e11708031be18a571ac9fc728960da099f3059e291b0\": container with ID starting with 78fb1eb44e00a7f81924e11708031be18a571ac9fc728960da099f3059e291b0 not found: ID does not exist" Jul 08 17:46:32.189376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.189316 2584 scope.go:117] "RemoveContainer" containerID="9b84fc6056efc21477c57523cb5aa8d0f7278b6f72846fb4264aecd15160a993" Jul 08 17:46:32.197947 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.197928 2584 scope.go:117] "RemoveContainer" containerID="9b84fc6056efc21477c57523cb5aa8d0f7278b6f72846fb4264aecd15160a993" Jul 08 17:46:32.198196 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:46:32.198178 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b84fc6056efc21477c57523cb5aa8d0f7278b6f72846fb4264aecd15160a993\": container with ID starting with 9b84fc6056efc21477c57523cb5aa8d0f7278b6f72846fb4264aecd15160a993 not found: ID does not exist" containerID="9b84fc6056efc21477c57523cb5aa8d0f7278b6f72846fb4264aecd15160a993" Jul 08 17:46:32.198299 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.198203 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b84fc6056efc21477c57523cb5aa8d0f7278b6f72846fb4264aecd15160a993"} err="failed to get container status \"9b84fc6056efc21477c57523cb5aa8d0f7278b6f72846fb4264aecd15160a993\": rpc error: code = NotFound desc = could not find container \"9b84fc6056efc21477c57523cb5aa8d0f7278b6f72846fb4264aecd15160a993\": container with ID starting with 9b84fc6056efc21477c57523cb5aa8d0f7278b6f72846fb4264aecd15160a993 not found: ID does not exist" Jul 08 17:46:32.199547 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:46:32.199522 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sq8fl\" is forbidden: User \"system:node:ip-10-0-129-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-226.ec2.internal' and this object" podUID="cac6c36b-20e4-463f-93bf-694c0aed137f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" Jul 08 17:46:32.200013 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.199974 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9js7v" podStartSLOduration=1.664246422 podStartE2EDuration="25.199961116s" totalImagesPullingTime="23.535714694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:46:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:46:32.197341605 +0000 UTC m=+467.881438882" watchObservedRunningTime="2026-07-08 17:46:32.199961116 +0000 UTC m=+467.884058393" Jul 08 17:46:32.201469 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:46:32.201448 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-hfsqj\" is forbidden: User \"system:node:ip-10-0-129-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-226.ec2.internal' and this object" podUID="40edb0c7-4add-45b6-8d93-f05753b8a2e8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" Jul 08 17:46:32.203403 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:46:32.203383 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sq8fl\" is forbidden: User \"system:node:ip-10-0-129-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-226.ec2.internal' and this object" podUID="cac6c36b-20e4-463f-93bf-694c0aed137f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sq8fl" Jul 08 17:46:32.224821 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:46:32.224785 2584 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-hfsqj\" is forbidden: User \"system:node:ip-10-0-129-226.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-226.ec2.internal' and this object" podUID="40edb0c7-4add-45b6-8d93-f05753b8a2e8" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfsqj" Jul 08 17:46:32.225131 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.225093 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg" podStartSLOduration=15.225080309 podStartE2EDuration="15.225080309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:46:32.222759963 +0000 UTC m=+467.906857241" watchObservedRunningTime="2026-07-08 17:46:32.225080309 +0000 UTC m=+467.909177586" Jul 08 17:46:32.244293 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.244219 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rd5p5" podStartSLOduration=15.244202634 podStartE2EDuration="15.244202634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:46:32.24196478 +0000 UTC m=+467.926062071" watchObservedRunningTime="2026-07-08 17:46:32.244202634 +0000 UTC m=+467.928299957" Jul 08 17:46:32.956563 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.956482 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40edb0c7-4add-45b6-8d93-f05753b8a2e8" path="/var/lib/kubelet/pods/40edb0c7-4add-45b6-8d93-f05753b8a2e8/volumes" Jul 08 17:46:32.956816 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:32.956804 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cac6c36b-20e4-463f-93bf-694c0aed137f" path="/var/lib/kubelet/pods/cac6c36b-20e4-463f-93bf-694c0aed137f/volumes" Jul 08 17:46:33.904590 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:33.904539 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg"] Jul 08 17:46:34.109683 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:34.109650 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz"] Jul 08 17:46:34.114754 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:34.114737 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz" Jul 08 17:46:34.127847 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:34.127824 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz"] Jul 08 17:46:34.198836 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:34.198731 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg" podUID="21d3761b-7a6a-413f-92d4-13ac0e6e042a" containerName="manager" containerID="cri-o://8a97e2c63c806bf5825c7ec33c3437673e6a64846689edd20845c56786bd66eb" gracePeriod=10 Jul 08 17:46:34.302482 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:34.302447 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b27da8f8-ac5e-4c39-ae38-10575f4548a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mjqbz\" (UID: \"b27da8f8-ac5e-4c39-ae38-10575f4548a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz" Jul 08 17:46:34.302682 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:34.302497 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxbhc\" (UniqueName: \"kubernetes.io/projected/b27da8f8-ac5e-4c39-ae38-10575f4548a1-kube-api-access-lxbhc\") pod \"kuadrant-operator-controller-manager-86878c769f-mjqbz\" (UID: \"b27da8f8-ac5e-4c39-ae38-10575f4548a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz" Jul 08 17:46:34.403912 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:34.403881 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b27da8f8-ac5e-4c39-ae38-10575f4548a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mjqbz\" (UID: \"b27da8f8-ac5e-4c39-ae38-10575f4548a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz" Jul 08 17:46:34.404058 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:34.403923 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxbhc\" (UniqueName: \"kubernetes.io/projected/b27da8f8-ac5e-4c39-ae38-10575f4548a1-kube-api-access-lxbhc\") pod \"kuadrant-operator-controller-manager-86878c769f-mjqbz\" (UID: \"b27da8f8-ac5e-4c39-ae38-10575f4548a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz" Jul 08 17:46:34.404304 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:34.404282 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b27da8f8-ac5e-4c39-ae38-10575f4548a1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-mjqbz\" (UID: \"b27da8f8-ac5e-4c39-ae38-10575f4548a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz" Jul 08 17:46:34.415123 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:34.415092 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxbhc\" (UniqueName: \"kubernetes.io/projected/b27da8f8-ac5e-4c39-ae38-10575f4548a1-kube-api-access-lxbhc\") pod \"kuadrant-operator-controller-manager-86878c769f-mjqbz\" (UID: \"b27da8f8-ac5e-4c39-ae38-10575f4548a1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz" Jul 08 17:46:34.424738 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:34.424715 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz" Jul 08 17:46:34.463431 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:34.463393 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg" Jul 08 17:46:34.574448 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:34.574422 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz"] Jul 08 17:46:34.576785 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:46:34.576753 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27da8f8_ac5e_4c39_ae38_10575f4548a1.slice/crio-bdd7e543c1566aa686250257e064ded2e2a22643647b46fa590e8dc3d421a021 WatchSource:0}: Error finding container bdd7e543c1566aa686250257e064ded2e2a22643647b46fa590e8dc3d421a021: Status 404 returned error can't find the container with id bdd7e543c1566aa686250257e064ded2e2a22643647b46fa590e8dc3d421a021 Jul 08 17:46:34.605043 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:34.605014 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/21d3761b-7a6a-413f-92d4-13ac0e6e042a-extensions-socket-volume\") pod \"21d3761b-7a6a-413f-92d4-13ac0e6e042a\" (UID: \"21d3761b-7a6a-413f-92d4-13ac0e6e042a\") " Jul 08 17:46:34.605185 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:34.605070 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzsp5\" (UniqueName: \"kubernetes.io/projected/21d3761b-7a6a-413f-92d4-13ac0e6e042a-kube-api-access-hzsp5\") pod \"21d3761b-7a6a-413f-92d4-13ac0e6e042a\" (UID: \"21d3761b-7a6a-413f-92d4-13ac0e6e042a\") " Jul 08 17:46:34.605428 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:34.605404 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d3761b-7a6a-413f-92d4-13ac0e6e042a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "21d3761b-7a6a-413f-92d4-13ac0e6e042a" (UID: "21d3761b-7a6a-413f-92d4-13ac0e6e042a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:46:34.607066 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:34.607046 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d3761b-7a6a-413f-92d4-13ac0e6e042a-kube-api-access-hzsp5" (OuterVolumeSpecName: "kube-api-access-hzsp5") pod "21d3761b-7a6a-413f-92d4-13ac0e6e042a" (UID: "21d3761b-7a6a-413f-92d4-13ac0e6e042a"). InnerVolumeSpecName "kube-api-access-hzsp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:46:34.706028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:34.705938 2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/21d3761b-7a6a-413f-92d4-13ac0e6e042a-extensions-socket-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:46:34.706028 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:34.705969 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzsp5\" (UniqueName: \"kubernetes.io/projected/21d3761b-7a6a-413f-92d4-13ac0e6e042a-kube-api-access-hzsp5\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:46:35.206046 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:35.206013 2584 generic.go:356] "Generic (PLEG): container finished" podID="21d3761b-7a6a-413f-92d4-13ac0e6e042a" containerID="8a97e2c63c806bf5825c7ec33c3437673e6a64846689edd20845c56786bd66eb" exitCode=0 Jul 08 17:46:35.206492 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:35.206057 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg" Jul 08 17:46:35.206492 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:35.206108 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg" event={"ID":"21d3761b-7a6a-413f-92d4-13ac0e6e042a","Type":"ContainerDied","Data":"8a97e2c63c806bf5825c7ec33c3437673e6a64846689edd20845c56786bd66eb"} Jul 08 17:46:35.206492 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:35.206165 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg" event={"ID":"21d3761b-7a6a-413f-92d4-13ac0e6e042a","Type":"ContainerDied","Data":"16985b3e9dc927d88e82ce1bfcb225b0f48e37514a790f216c185b267f51be50"} Jul 08 17:46:35.206492 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:35.206190 2584 scope.go:117] "RemoveContainer" containerID="8a97e2c63c806bf5825c7ec33c3437673e6a64846689edd20845c56786bd66eb" Jul 08 17:46:35.208110 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:35.208085 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz" event={"ID":"b27da8f8-ac5e-4c39-ae38-10575f4548a1","Type":"ContainerStarted","Data":"75ba9ab7fd060a5e0577d2522926b847ed3c09381285d38172c1421c2864361e"} Jul 08 17:46:35.208222 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:35.208138 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz" event={"ID":"b27da8f8-ac5e-4c39-ae38-10575f4548a1","Type":"ContainerStarted","Data":"bdd7e543c1566aa686250257e064ded2e2a22643647b46fa590e8dc3d421a021"} Jul 08 17:46:35.208222 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:35.208173 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz" Jul 08 17:46:35.222099 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:35.222078 2584 scope.go:117] "RemoveContainer" containerID="8a97e2c63c806bf5825c7ec33c3437673e6a64846689edd20845c56786bd66eb" Jul 08 17:46:35.222418 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:46:35.222390 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a97e2c63c806bf5825c7ec33c3437673e6a64846689edd20845c56786bd66eb\": container with ID starting with 8a97e2c63c806bf5825c7ec33c3437673e6a64846689edd20845c56786bd66eb not found: ID does not exist" containerID="8a97e2c63c806bf5825c7ec33c3437673e6a64846689edd20845c56786bd66eb" Jul 08 17:46:35.222520 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:35.222423 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a97e2c63c806bf5825c7ec33c3437673e6a64846689edd20845c56786bd66eb"} err="failed to get container status \"8a97e2c63c806bf5825c7ec33c3437673e6a64846689edd20845c56786bd66eb\": rpc error: code = NotFound desc = could not find container \"8a97e2c63c806bf5825c7ec33c3437673e6a64846689edd20845c56786bd66eb\": container with ID starting with 8a97e2c63c806bf5825c7ec33c3437673e6a64846689edd20845c56786bd66eb not found: ID does not exist" Jul 08 17:46:35.234681 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:35.234618 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz" podStartSLOduration=1.234592938 podStartE2EDuration="1.234592938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:46:35.230067025 +0000 UTC m=+470.914164314" watchObservedRunningTime="2026-07-08 17:46:35.234592938 +0000 UTC m=+470.918690221" Jul 08 17:46:35.252747 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:35.252710 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg"] Jul 08 17:46:35.255197 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:35.255168 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-27stg"] Jul 08 17:46:36.955938 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:36.955905 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21d3761b-7a6a-413f-92d4-13ac0e6e042a" path="/var/lib/kubelet/pods/21d3761b-7a6a-413f-92d4-13ac0e6e042a/volumes" Jul 08 17:46:43.195902 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:43.195867 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-rd5p5" Jul 08 17:46:46.217986 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:46.217957 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz" Jul 08 17:46:50.304570 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:50.304536 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:50.762556 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:50.762518 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:52.935050 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:52.935014 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:46:52.990650 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:46:52.990616 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:05.512164 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:05.512125 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:05.522418 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:05.522387 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n9772"] Jul 08 17:47:05.522774 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:05.522758 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21d3761b-7a6a-413f-92d4-13ac0e6e042a" containerName="manager" Jul 08 17:47:05.522774 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:05.522774 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="21d3761b-7a6a-413f-92d4-13ac0e6e042a" containerName="manager" Jul 08 17:47:05.522939 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:05.522863 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="21d3761b-7a6a-413f-92d4-13ac0e6e042a" containerName="manager" Jul 08 17:47:05.573476 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:05.573439 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n9772"] Jul 08 17:47:05.573636 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:05.573572 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-n9772" Jul 08 17:47:05.576083 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:05.576061 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 08 17:47:05.620276 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:05.620227 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n9772"] Jul 08 17:47:05.621366 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:05.621328 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:05.664589 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:05.664550 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e23445f5-a72d-4327-9068-51f9209adadd-config-file\") pod \"limitador-limitador-64c94fd64f-n9772\" (UID: \"e23445f5-a72d-4327-9068-51f9209adadd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n9772" Jul 08 17:47:05.664751 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:05.664597 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mn9w\" (UniqueName: \"kubernetes.io/projected/e23445f5-a72d-4327-9068-51f9209adadd-kube-api-access-4mn9w\") pod \"limitador-limitador-64c94fd64f-n9772\" (UID: \"e23445f5-a72d-4327-9068-51f9209adadd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n9772" Jul 08 17:47:05.686030 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:05.685990 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:05.765599 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:05.765517 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mn9w\" (UniqueName: \"kubernetes.io/projected/e23445f5-a72d-4327-9068-51f9209adadd-kube-api-access-4mn9w\") pod \"limitador-limitador-64c94fd64f-n9772\" (UID: \"e23445f5-a72d-4327-9068-51f9209adadd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n9772" Jul 08 17:47:05.765791 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:05.765615 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e23445f5-a72d-4327-9068-51f9209adadd-config-file\") pod \"limitador-limitador-64c94fd64f-n9772\" (UID: \"e23445f5-a72d-4327-9068-51f9209adadd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n9772" Jul 08 17:47:05.766295 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:05.766275 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e23445f5-a72d-4327-9068-51f9209adadd-config-file\") pod \"limitador-limitador-64c94fd64f-n9772\" (UID: \"e23445f5-a72d-4327-9068-51f9209adadd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n9772" Jul 08 17:47:05.774963 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:05.774925 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mn9w\" (UniqueName: \"kubernetes.io/projected/e23445f5-a72d-4327-9068-51f9209adadd-kube-api-access-4mn9w\") pod \"limitador-limitador-64c94fd64f-n9772\" (UID: \"e23445f5-a72d-4327-9068-51f9209adadd\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n9772" Jul 08 17:47:05.885802 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:05.885749 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-n9772" Jul 08 17:47:06.028756 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:06.028722 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n9772"] Jul 08 17:47:06.030185 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:47:06.030154 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23445f5_a72d_4327_9068_51f9209adadd.slice/crio-d69f35a241d87cb543763a948e31d0a8937cb1378373846fe7ea6079db4df259 WatchSource:0}: Error finding container d69f35a241d87cb543763a948e31d0a8937cb1378373846fe7ea6079db4df259: Status 404 returned error can't find the container with id d69f35a241d87cb543763a948e31d0a8937cb1378373846fe7ea6079db4df259 Jul 08 17:47:06.381348 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:06.381310 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-g87pv"] Jul 08 17:47:06.386647 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:06.386624 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g87pv" Jul 08 17:47:06.388821 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:06.388800 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mlx7t\"" Jul 08 17:47:06.389049 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:06.389026 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-n9772" event={"ID":"e23445f5-a72d-4327-9068-51f9209adadd","Type":"ContainerStarted","Data":"d69f35a241d87cb543763a948e31d0a8937cb1378373846fe7ea6079db4df259"} Jul 08 17:47:06.392046 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:06.392022 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-g87pv"] Jul 08 17:47:06.412946 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:06.412919 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:06.470916 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:06.470883 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc9pd\" (UniqueName: \"kubernetes.io/projected/e6785f03-e2ba-4be0-8a16-94b80b955a50-kube-api-access-hc9pd\") pod \"authorino-58cd955675-g87pv\" (UID: \"e6785f03-e2ba-4be0-8a16-94b80b955a50\") " pod="kuadrant-system/authorino-58cd955675-g87pv" Jul 08 17:47:06.516231 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:06.516191 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:06.571647 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:06.571611 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc9pd\" (UniqueName: \"kubernetes.io/projected/e6785f03-e2ba-4be0-8a16-94b80b955a50-kube-api-access-hc9pd\") pod \"authorino-58cd955675-g87pv\" (UID: \"e6785f03-e2ba-4be0-8a16-94b80b955a50\") " pod="kuadrant-system/authorino-58cd955675-g87pv" Jul 08 17:47:06.580873 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:06.580849 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc9pd\" (UniqueName: \"kubernetes.io/projected/e6785f03-e2ba-4be0-8a16-94b80b955a50-kube-api-access-hc9pd\") pod \"authorino-58cd955675-g87pv\" (UID: \"e6785f03-e2ba-4be0-8a16-94b80b955a50\") " pod="kuadrant-system/authorino-58cd955675-g87pv" Jul 08 17:47:06.698027 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:06.697938 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g87pv" Jul 08 17:47:06.901633 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:06.901591 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-g87pv"] Jul 08 17:47:06.903194 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:47:06.903157 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6785f03_e2ba_4be0_8a16_94b80b955a50.slice/crio-f3bea1b0a91889f399dd01fd299f700b6d35f682ce84f289bc5efa8f1934cc78 WatchSource:0}: Error finding container f3bea1b0a91889f399dd01fd299f700b6d35f682ce84f289bc5efa8f1934cc78: Status 404 returned error can't find the container with id f3bea1b0a91889f399dd01fd299f700b6d35f682ce84f289bc5efa8f1934cc78 Jul 08 17:47:07.398287 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:07.398254 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g87pv" event={"ID":"e6785f03-e2ba-4be0-8a16-94b80b955a50","Type":"ContainerStarted","Data":"f3bea1b0a91889f399dd01fd299f700b6d35f682ce84f289bc5efa8f1934cc78"} Jul 08 17:47:11.427077 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:11.427036 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g87pv" event={"ID":"e6785f03-e2ba-4be0-8a16-94b80b955a50","Type":"ContainerStarted","Data":"12134daf2d4affb3d5d80c3bbdfd21dc0af363df6d699175aed32ad6a33f3c0d"} Jul 08 17:47:11.428676 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:11.428653 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-n9772" event={"ID":"e23445f5-a72d-4327-9068-51f9209adadd","Type":"ContainerStarted","Data":"c34ddbee2c23a6c1dfde2f1a6b9bf6d4ca40c3e074274212c5fa68e79aba8161"} Jul 08 17:47:11.428797 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:11.428787 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-n9772" Jul 08 17:47:11.444436 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:11.444383 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-g87pv" podStartSLOduration=1.6564516440000001 podStartE2EDuration="5.444367695s" totalImagesPullingTime="3.787916051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:47:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:47:11.442403565 +0000 UTC m=+507.126500842" watchObservedRunningTime="2026-07-08 17:47:11.444367695 +0000 UTC m=+507.128464987" Jul 08 17:47:12.176859 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:12.176796 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-n9772" podStartSLOduration=2.572908891 podStartE2EDuration="7.1767788s" totalImagesPullingTime="4.603869909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:47:11.459552843 +0000 UTC m=+507.143650115" watchObservedRunningTime="2026-07-08 17:47:12.1767788 +0000 UTC m=+507.860876078" Jul 08 17:47:12.177902 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:12.177882 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-g87pv"] Jul 08 17:47:13.439181 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:13.439142 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-g87pv" podUID="e6785f03-e2ba-4be0-8a16-94b80b955a50" containerName="authorino" containerID="cri-o://12134daf2d4affb3d5d80c3bbdfd21dc0af363df6d699175aed32ad6a33f3c0d" gracePeriod=30 Jul 08 17:47:13.692473 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:13.692406 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g87pv" Jul 08 17:47:13.838695 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:13.838653 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc9pd\" (UniqueName: \"kubernetes.io/projected/e6785f03-e2ba-4be0-8a16-94b80b955a50-kube-api-access-hc9pd\") pod \"e6785f03-e2ba-4be0-8a16-94b80b955a50\" (UID: \"e6785f03-e2ba-4be0-8a16-94b80b955a50\") " Jul 08 17:47:13.840747 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:13.840718 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6785f03-e2ba-4be0-8a16-94b80b955a50-kube-api-access-hc9pd" (OuterVolumeSpecName: "kube-api-access-hc9pd") pod "e6785f03-e2ba-4be0-8a16-94b80b955a50" (UID: "e6785f03-e2ba-4be0-8a16-94b80b955a50"). InnerVolumeSpecName "kube-api-access-hc9pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:47:13.939329 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:13.939295 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc9pd\" (UniqueName: \"kubernetes.io/projected/e6785f03-e2ba-4be0-8a16-94b80b955a50-kube-api-access-hc9pd\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:47:14.445994 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:14.445948 2584 generic.go:356] "Generic (PLEG): container finished" podID="e6785f03-e2ba-4be0-8a16-94b80b955a50" containerID="12134daf2d4affb3d5d80c3bbdfd21dc0af363df6d699175aed32ad6a33f3c0d" exitCode=0 Jul 08 17:47:14.445994 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:14.445996 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-g87pv" Jul 08 17:47:14.446567 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:14.446044 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g87pv" event={"ID":"e6785f03-e2ba-4be0-8a16-94b80b955a50","Type":"ContainerDied","Data":"12134daf2d4affb3d5d80c3bbdfd21dc0af363df6d699175aed32ad6a33f3c0d"} Jul 08 17:47:14.446567 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:14.446083 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-g87pv" event={"ID":"e6785f03-e2ba-4be0-8a16-94b80b955a50","Type":"ContainerDied","Data":"f3bea1b0a91889f399dd01fd299f700b6d35f682ce84f289bc5efa8f1934cc78"} Jul 08 17:47:14.446567 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:14.446103 2584 scope.go:117] "RemoveContainer" containerID="12134daf2d4affb3d5d80c3bbdfd21dc0af363df6d699175aed32ad6a33f3c0d" Jul 08 17:47:14.455269 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:14.455233 2584 scope.go:117] "RemoveContainer" containerID="12134daf2d4affb3d5d80c3bbdfd21dc0af363df6d699175aed32ad6a33f3c0d" Jul 08 17:47:14.455557 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:47:14.455538 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12134daf2d4affb3d5d80c3bbdfd21dc0af363df6d699175aed32ad6a33f3c0d\": container with ID starting with 12134daf2d4affb3d5d80c3bbdfd21dc0af363df6d699175aed32ad6a33f3c0d not found: ID does not exist" containerID="12134daf2d4affb3d5d80c3bbdfd21dc0af363df6d699175aed32ad6a33f3c0d" Jul 08 17:47:14.455620 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:14.455566 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12134daf2d4affb3d5d80c3bbdfd21dc0af363df6d699175aed32ad6a33f3c0d"} err="failed to get container status \"12134daf2d4affb3d5d80c3bbdfd21dc0af363df6d699175aed32ad6a33f3c0d\": rpc error: code = NotFound desc = could not find container \"12134daf2d4affb3d5d80c3bbdfd21dc0af363df6d699175aed32ad6a33f3c0d\": container with ID starting with 12134daf2d4affb3d5d80c3bbdfd21dc0af363df6d699175aed32ad6a33f3c0d not found: ID does not exist" Jul 08 17:47:14.469957 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:14.469922 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-g87pv"] Jul 08 17:47:14.475344 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:14.475317 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-g87pv"] Jul 08 17:47:14.956880 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:14.956841 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6785f03-e2ba-4be0-8a16-94b80b955a50" path="/var/lib/kubelet/pods/e6785f03-e2ba-4be0-8a16-94b80b955a50/volumes" Jul 08 17:47:19.470422 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:19.470384 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:22.435196 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:22.435168 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-n9772" Jul 08 17:47:23.141829 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:23.141793 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n9772"] Jul 08 17:47:23.142053 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:23.142026 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-n9772" podUID="e23445f5-a72d-4327-9068-51f9209adadd" containerName="limitador" containerID="cri-o://c34ddbee2c23a6c1dfde2f1a6b9bf6d4ca40c3e074274212c5fa68e79aba8161" gracePeriod=30 Jul 08 17:47:23.497628 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:23.497591 2584 generic.go:356] "Generic (PLEG): container finished" podID="e23445f5-a72d-4327-9068-51f9209adadd" containerID="c34ddbee2c23a6c1dfde2f1a6b9bf6d4ca40c3e074274212c5fa68e79aba8161" exitCode=0 Jul 08 17:47:23.497981 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:23.497652 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-n9772" event={"ID":"e23445f5-a72d-4327-9068-51f9209adadd","Type":"ContainerDied","Data":"c34ddbee2c23a6c1dfde2f1a6b9bf6d4ca40c3e074274212c5fa68e79aba8161"} Jul 08 17:47:24.090394 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:24.090369 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-n9772" Jul 08 17:47:24.228575 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:24.228459 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mn9w\" (UniqueName: \"kubernetes.io/projected/e23445f5-a72d-4327-9068-51f9209adadd-kube-api-access-4mn9w\") pod \"e23445f5-a72d-4327-9068-51f9209adadd\" (UID: \"e23445f5-a72d-4327-9068-51f9209adadd\") " Jul 08 17:47:24.228575 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:24.228553 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e23445f5-a72d-4327-9068-51f9209adadd-config-file\") pod \"e23445f5-a72d-4327-9068-51f9209adadd\" (UID: \"e23445f5-a72d-4327-9068-51f9209adadd\") " Jul 08 17:47:24.228917 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:24.228891 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23445f5-a72d-4327-9068-51f9209adadd-config-file" (OuterVolumeSpecName: "config-file") pod "e23445f5-a72d-4327-9068-51f9209adadd" (UID: "e23445f5-a72d-4327-9068-51f9209adadd"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 17:47:24.230586 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:24.230563 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23445f5-a72d-4327-9068-51f9209adadd-kube-api-access-4mn9w" (OuterVolumeSpecName: "kube-api-access-4mn9w") pod "e23445f5-a72d-4327-9068-51f9209adadd" (UID: "e23445f5-a72d-4327-9068-51f9209adadd"). InnerVolumeSpecName "kube-api-access-4mn9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:47:24.329411 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:24.329361 2584 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e23445f5-a72d-4327-9068-51f9209adadd-config-file\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:47:24.329411 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:24.329407 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mn9w\" (UniqueName: \"kubernetes.io/projected/e23445f5-a72d-4327-9068-51f9209adadd-kube-api-access-4mn9w\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:47:24.504683 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:24.504602 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-n9772" Jul 08 17:47:24.504683 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:24.504620 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-n9772" event={"ID":"e23445f5-a72d-4327-9068-51f9209adadd","Type":"ContainerDied","Data":"d69f35a241d87cb543763a948e31d0a8937cb1378373846fe7ea6079db4df259"} Jul 08 17:47:24.504683 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:24.504675 2584 scope.go:117] "RemoveContainer" containerID="c34ddbee2c23a6c1dfde2f1a6b9bf6d4ca40c3e074274212c5fa68e79aba8161" Jul 08 17:47:24.528621 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:24.528332 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n9772"] Jul 08 17:47:24.530676 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:24.530658 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n9772"] Jul 08 17:47:24.957213 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:24.957178 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e23445f5-a72d-4327-9068-51f9209adadd" path="/var/lib/kubelet/pods/e23445f5-a72d-4327-9068-51f9209adadd/volumes" Jul 08 17:47:29.036644 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.036601 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:29.131982 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.131933 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:29.507911 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.507868 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4fqdv"] Jul 08 17:47:29.508424 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.508408 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6785f03-e2ba-4be0-8a16-94b80b955a50" containerName="authorino" Jul 08 17:47:29.508478 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.508431 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6785f03-e2ba-4be0-8a16-94b80b955a50" containerName="authorino" Jul 08 17:47:29.508478 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.508461 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e23445f5-a72d-4327-9068-51f9209adadd" containerName="limitador" Jul 08 17:47:29.508478 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.508472 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e23445f5-a72d-4327-9068-51f9209adadd" containerName="limitador" Jul 08 17:47:29.508575 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.508566 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6785f03-e2ba-4be0-8a16-94b80b955a50" containerName="authorino" Jul 08 17:47:29.508614 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.508578 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e23445f5-a72d-4327-9068-51f9209adadd" containerName="limitador" Jul 08 17:47:29.513360 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.513337 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4fqdv" Jul 08 17:47:29.515972 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.515949 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 17:47:29.516089 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.516031 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 08 17:47:29.516141 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.516091 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 17:47:29.516634 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.516616 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-rc2mr\"" Jul 08 17:47:29.522083 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.522058 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4fqdv"] Jul 08 17:47:29.541224 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.541191 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:29.576047 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.576010 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2krj\" (UniqueName: \"kubernetes.io/projected/48631804-ad4c-42dc-9c7f-d25bf5d92ebd-kube-api-access-w2krj\") pod \"postgres-69fb4594cf-4fqdv\" (UID: \"48631804-ad4c-42dc-9c7f-d25bf5d92ebd\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4fqdv" Jul 08 17:47:29.576047 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.576049 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/48631804-ad4c-42dc-9c7f-d25bf5d92ebd-data\") pod \"postgres-69fb4594cf-4fqdv\" (UID: \"48631804-ad4c-42dc-9c7f-d25bf5d92ebd\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4fqdv" Jul 08 17:47:29.677531 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.677493 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2krj\" (UniqueName: \"kubernetes.io/projected/48631804-ad4c-42dc-9c7f-d25bf5d92ebd-kube-api-access-w2krj\") pod \"postgres-69fb4594cf-4fqdv\" (UID: \"48631804-ad4c-42dc-9c7f-d25bf5d92ebd\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4fqdv" Jul 08 17:47:29.677531 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.677535 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/48631804-ad4c-42dc-9c7f-d25bf5d92ebd-data\") pod \"postgres-69fb4594cf-4fqdv\" (UID: \"48631804-ad4c-42dc-9c7f-d25bf5d92ebd\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4fqdv" Jul 08 17:47:29.677921 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.677903 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/48631804-ad4c-42dc-9c7f-d25bf5d92ebd-data\") pod \"postgres-69fb4594cf-4fqdv\" (UID: \"48631804-ad4c-42dc-9c7f-d25bf5d92ebd\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4fqdv" Jul 08 17:47:29.687292 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.687262 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2krj\" (UniqueName: \"kubernetes.io/projected/48631804-ad4c-42dc-9c7f-d25bf5d92ebd-kube-api-access-w2krj\") pod \"postgres-69fb4594cf-4fqdv\" (UID: \"48631804-ad4c-42dc-9c7f-d25bf5d92ebd\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4fqdv" Jul 08 17:47:29.825318 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.825226 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4fqdv" Jul 08 17:47:29.963517 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:29.963490 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4fqdv"] Jul 08 17:47:29.965543 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:47:29.965515 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48631804_ad4c_42dc_9c7f_d25bf5d92ebd.slice/crio-69968116a32e4ebea9a5a5478f4c763a33264dd8d2b7364c6fcb48133bcadcf5 WatchSource:0}: Error finding container 69968116a32e4ebea9a5a5478f4c763a33264dd8d2b7364c6fcb48133bcadcf5: Status 404 returned error can't find the container with id 69968116a32e4ebea9a5a5478f4c763a33264dd8d2b7364c6fcb48133bcadcf5 Jul 08 17:47:30.547944 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:30.547913 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4fqdv" event={"ID":"48631804-ad4c-42dc-9c7f-d25bf5d92ebd","Type":"ContainerStarted","Data":"69968116a32e4ebea9a5a5478f4c763a33264dd8d2b7364c6fcb48133bcadcf5"} Jul 08 17:47:35.580786 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:35.580742 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4fqdv" event={"ID":"48631804-ad4c-42dc-9c7f-d25bf5d92ebd","Type":"ContainerStarted","Data":"2fdd22e4beda7569c8557cbb405d82385e63c877912f72e0e297764bde712e4c"} Jul 08 17:47:35.581164 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:35.580865 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4fqdv" Jul 08 17:47:35.598903 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:35.598847 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4fqdv" podStartSLOduration=1.637643637 podStartE2EDuration="6.598831922s" totalImagesPullingTime="4.961188285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:47:35.59851092 +0000 UTC m=+531.282608199" watchObservedRunningTime="2026-07-08 17:47:35.598831922 +0000 UTC m=+531.282929198" Jul 08 17:47:41.615928 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:41.615851 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4fqdv" Jul 08 17:47:42.175454 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.175417 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jjcc6"] Jul 08 17:47:42.249109 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.249067 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jjcc6"] Jul 08 17:47:42.249296 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.249191 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jjcc6" Jul 08 17:47:42.252221 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.252198 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mlx7t\"" Jul 08 17:47:42.287617 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.287580 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf7j6\" (UniqueName: \"kubernetes.io/projected/8fe1c948-c788-45a3-9fd4-f2f86db800a9-kube-api-access-jf7j6\") pod \"authorino-5d4b67c446-jjcc6\" (UID: \"8fe1c948-c788-45a3-9fd4-f2f86db800a9\") " pod="kuadrant-system/authorino-5d4b67c446-jjcc6" Jul 08 17:47:42.289097 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.289068 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:42.366313 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.366231 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:42.388472 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.388434 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf7j6\" (UniqueName: \"kubernetes.io/projected/8fe1c948-c788-45a3-9fd4-f2f86db800a9-kube-api-access-jf7j6\") pod \"authorino-5d4b67c446-jjcc6\" (UID: \"8fe1c948-c788-45a3-9fd4-f2f86db800a9\") " pod="kuadrant-system/authorino-5d4b67c446-jjcc6" Jul 08 17:47:42.413964 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.413930 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf7j6\" (UniqueName: \"kubernetes.io/projected/8fe1c948-c788-45a3-9fd4-f2f86db800a9-kube-api-access-jf7j6\") pod \"authorino-5d4b67c446-jjcc6\" (UID: \"8fe1c948-c788-45a3-9fd4-f2f86db800a9\") " pod="kuadrant-system/authorino-5d4b67c446-jjcc6" Jul 08 17:47:42.427670 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.427602 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jjcc6"] Jul 08 17:47:42.427867 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.427855 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jjcc6" Jul 08 17:47:42.457400 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.457373 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-774dfc4d85-kwp77"] Jul 08 17:47:42.484104 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.484067 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-774dfc4d85-kwp77"] Jul 08 17:47:42.484104 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.484101 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-774dfc4d85-kwp77" Jul 08 17:47:42.585217 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.585189 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jjcc6"] Jul 08 17:47:42.586988 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:47:42.586952 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fe1c948_c788_45a3_9fd4_f2f86db800a9.slice/crio-34e48c118159f4f9e24ed7963e2305d2bdc2ebf3f567db4ffe41be20c5bed648 WatchSource:0}: Error finding container 34e48c118159f4f9e24ed7963e2305d2bdc2ebf3f567db4ffe41be20c5bed648: Status 404 returned error can't find the container with id 34e48c118159f4f9e24ed7963e2305d2bdc2ebf3f567db4ffe41be20c5bed648 Jul 08 17:47:42.590272 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.590206 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbzvp\" (UniqueName: \"kubernetes.io/projected/75a54d6d-8d0f-404e-af7a-449464111dc5-kube-api-access-jbzvp\") pod \"authorino-774dfc4d85-kwp77\" (UID: \"75a54d6d-8d0f-404e-af7a-449464111dc5\") " pod="kuadrant-system/authorino-774dfc4d85-kwp77" Jul 08 17:47:42.618356 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.618333 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jjcc6" event={"ID":"8fe1c948-c788-45a3-9fd4-f2f86db800a9","Type":"ContainerStarted","Data":"34e48c118159f4f9e24ed7963e2305d2bdc2ebf3f567db4ffe41be20c5bed648"} Jul 08 17:47:42.691206 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.691119 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbzvp\" (UniqueName: \"kubernetes.io/projected/75a54d6d-8d0f-404e-af7a-449464111dc5-kube-api-access-jbzvp\") pod \"authorino-774dfc4d85-kwp77\" (UID: \"75a54d6d-8d0f-404e-af7a-449464111dc5\") " pod="kuadrant-system/authorino-774dfc4d85-kwp77" Jul 08 17:47:42.700713 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.700686 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbzvp\" (UniqueName: \"kubernetes.io/projected/75a54d6d-8d0f-404e-af7a-449464111dc5-kube-api-access-jbzvp\") pod \"authorino-774dfc4d85-kwp77\" (UID: \"75a54d6d-8d0f-404e-af7a-449464111dc5\") " pod="kuadrant-system/authorino-774dfc4d85-kwp77" Jul 08 17:47:42.746143 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.746101 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-774dfc4d85-kwp77"] Jul 08 17:47:42.746394 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.746381 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-774dfc4d85-kwp77" Jul 08 17:47:42.776850 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.776820 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68979f9cc7-wfl5g"] Jul 08 17:47:42.779675 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.779653 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68979f9cc7-wfl5g" Jul 08 17:47:42.782117 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.782096 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 08 17:47:42.788457 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.788435 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68979f9cc7-wfl5g"] Jul 08 17:47:42.884954 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.884927 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-774dfc4d85-kwp77"] Jul 08 17:47:42.886588 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:47:42.886553 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75a54d6d_8d0f_404e_af7a_449464111dc5.slice/crio-9f9d1bf63c022e24e8aaefb4f631e9b21568d7ef19a13da3bfb8de402eccfa64 WatchSource:0}: Error finding container 9f9d1bf63c022e24e8aaefb4f631e9b21568d7ef19a13da3bfb8de402eccfa64: Status 404 returned error can't find the container with id 9f9d1bf63c022e24e8aaefb4f631e9b21568d7ef19a13da3bfb8de402eccfa64 Jul 08 17:47:42.893277 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.893225 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e1775e1d-763f-482a-b327-da8ad008d9be-tls-cert\") pod \"authorino-68979f9cc7-wfl5g\" (UID: \"e1775e1d-763f-482a-b327-da8ad008d9be\") " pod="kuadrant-system/authorino-68979f9cc7-wfl5g" Jul 08 17:47:42.893376 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.893354 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vckf2\" (UniqueName: \"kubernetes.io/projected/e1775e1d-763f-482a-b327-da8ad008d9be-kube-api-access-vckf2\") pod \"authorino-68979f9cc7-wfl5g\" (UID: \"e1775e1d-763f-482a-b327-da8ad008d9be\") " pod="kuadrant-system/authorino-68979f9cc7-wfl5g" Jul 08 17:47:42.994470 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.994379 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vckf2\" (UniqueName: \"kubernetes.io/projected/e1775e1d-763f-482a-b327-da8ad008d9be-kube-api-access-vckf2\") pod \"authorino-68979f9cc7-wfl5g\" (UID: \"e1775e1d-763f-482a-b327-da8ad008d9be\") " pod="kuadrant-system/authorino-68979f9cc7-wfl5g" Jul 08 17:47:42.994470 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.994432 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e1775e1d-763f-482a-b327-da8ad008d9be-tls-cert\") pod \"authorino-68979f9cc7-wfl5g\" (UID: \"e1775e1d-763f-482a-b327-da8ad008d9be\") " pod="kuadrant-system/authorino-68979f9cc7-wfl5g" Jul 08 17:47:42.996844 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:42.996825 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e1775e1d-763f-482a-b327-da8ad008d9be-tls-cert\") pod \"authorino-68979f9cc7-wfl5g\" (UID: \"e1775e1d-763f-482a-b327-da8ad008d9be\") " pod="kuadrant-system/authorino-68979f9cc7-wfl5g" Jul 08 17:47:43.003517 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:43.003490 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vckf2\" (UniqueName: \"kubernetes.io/projected/e1775e1d-763f-482a-b327-da8ad008d9be-kube-api-access-vckf2\") pod \"authorino-68979f9cc7-wfl5g\" (UID: \"e1775e1d-763f-482a-b327-da8ad008d9be\") " pod="kuadrant-system/authorino-68979f9cc7-wfl5g" Jul 08 17:47:43.090751 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:43.090713 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68979f9cc7-wfl5g" Jul 08 17:47:43.229407 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:43.229380 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68979f9cc7-wfl5g"] Jul 08 17:47:43.230571 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:47:43.230544 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1775e1d_763f_482a_b327_da8ad008d9be.slice/crio-b5467df89163757fdd323daa13619f3d1795c2a19405343e01a3b350f9dde2c5 WatchSource:0}: Error finding container b5467df89163757fdd323daa13619f3d1795c2a19405343e01a3b350f9dde2c5: Status 404 returned error can't find the container with id b5467df89163757fdd323daa13619f3d1795c2a19405343e01a3b350f9dde2c5 Jul 08 17:47:43.625665 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:43.625633 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-774dfc4d85-kwp77" event={"ID":"75a54d6d-8d0f-404e-af7a-449464111dc5","Type":"ContainerStarted","Data":"2705da7bacdfbd87b76c920a1da8a990102008171cfab60f44025bbdd0d27381"} Jul 08 17:47:43.626064 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:43.625682 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-774dfc4d85-kwp77" event={"ID":"75a54d6d-8d0f-404e-af7a-449464111dc5","Type":"ContainerStarted","Data":"9f9d1bf63c022e24e8aaefb4f631e9b21568d7ef19a13da3bfb8de402eccfa64"} Jul 08 17:47:43.626064 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:43.625684 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-774dfc4d85-kwp77" podUID="75a54d6d-8d0f-404e-af7a-449464111dc5" containerName="authorino" containerID="cri-o://2705da7bacdfbd87b76c920a1da8a990102008171cfab60f44025bbdd0d27381" gracePeriod=30 Jul 08 17:47:43.627019 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:43.626998 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68979f9cc7-wfl5g" event={"ID":"e1775e1d-763f-482a-b327-da8ad008d9be","Type":"ContainerStarted","Data":"b5467df89163757fdd323daa13619f3d1795c2a19405343e01a3b350f9dde2c5"} Jul 08 17:47:43.643605 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:43.643559 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-774dfc4d85-kwp77" podStartSLOduration=1.085740868 podStartE2EDuration="1.643542519s" totalImagesPullingTime="557.801651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:47:43.642621146 +0000 UTC m=+539.326718424" watchObservedRunningTime="2026-07-08 17:47:43.643542519 +0000 UTC m=+539.327639796" Jul 08 17:47:43.889069 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:43.889043 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-774dfc4d85-kwp77" Jul 08 17:47:44.002972 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:44.002852 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbzvp\" (UniqueName: \"kubernetes.io/projected/75a54d6d-8d0f-404e-af7a-449464111dc5-kube-api-access-jbzvp\") pod \"75a54d6d-8d0f-404e-af7a-449464111dc5\" (UID: \"75a54d6d-8d0f-404e-af7a-449464111dc5\") " Jul 08 17:47:44.004898 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:44.004873 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a54d6d-8d0f-404e-af7a-449464111dc5-kube-api-access-jbzvp" (OuterVolumeSpecName: "kube-api-access-jbzvp") pod "75a54d6d-8d0f-404e-af7a-449464111dc5" (UID: "75a54d6d-8d0f-404e-af7a-449464111dc5"). InnerVolumeSpecName "kube-api-access-jbzvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:47:44.104145 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:44.104097 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbzvp\" (UniqueName: \"kubernetes.io/projected/75a54d6d-8d0f-404e-af7a-449464111dc5-kube-api-access-jbzvp\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:47:44.635193 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:44.635150 2584 generic.go:356] "Generic (PLEG): container finished" podID="75a54d6d-8d0f-404e-af7a-449464111dc5" containerID="2705da7bacdfbd87b76c920a1da8a990102008171cfab60f44025bbdd0d27381" exitCode=0 Jul 08 17:47:44.635652 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:44.635211 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-774dfc4d85-kwp77" Jul 08 17:47:44.635652 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:44.635256 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-774dfc4d85-kwp77" event={"ID":"75a54d6d-8d0f-404e-af7a-449464111dc5","Type":"ContainerDied","Data":"2705da7bacdfbd87b76c920a1da8a990102008171cfab60f44025bbdd0d27381"} Jul 08 17:47:44.635652 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:44.635302 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-774dfc4d85-kwp77" event={"ID":"75a54d6d-8d0f-404e-af7a-449464111dc5","Type":"ContainerDied","Data":"9f9d1bf63c022e24e8aaefb4f631e9b21568d7ef19a13da3bfb8de402eccfa64"} Jul 08 17:47:44.635652 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:44.635320 2584 scope.go:117] "RemoveContainer" containerID="2705da7bacdfbd87b76c920a1da8a990102008171cfab60f44025bbdd0d27381" Jul 08 17:47:44.636953 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:44.636923 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jjcc6" event={"ID":"8fe1c948-c788-45a3-9fd4-f2f86db800a9","Type":"ContainerStarted","Data":"06827b981617367547e108621d18677a7617c700450a624acc112103cea46555"} Jul 08 17:47:44.637083 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:44.637001 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-jjcc6" podUID="8fe1c948-c788-45a3-9fd4-f2f86db800a9" containerName="authorino" containerID="cri-o://06827b981617367547e108621d18677a7617c700450a624acc112103cea46555" gracePeriod=30 Jul 08 17:47:44.638757 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:44.638726 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68979f9cc7-wfl5g" event={"ID":"e1775e1d-763f-482a-b327-da8ad008d9be","Type":"ContainerStarted","Data":"162ad1e65ef769be12f0a011c4c3de02bb2acc347f9fb301ef0a3c40cb1422eb"} Jul 08 17:47:44.646463 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:44.646444 2584 scope.go:117] "RemoveContainer" containerID="2705da7bacdfbd87b76c920a1da8a990102008171cfab60f44025bbdd0d27381" Jul 08 17:47:44.646727 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:47:44.646707 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2705da7bacdfbd87b76c920a1da8a990102008171cfab60f44025bbdd0d27381\": container with ID starting with 2705da7bacdfbd87b76c920a1da8a990102008171cfab60f44025bbdd0d27381 not found: ID does not exist" containerID="2705da7bacdfbd87b76c920a1da8a990102008171cfab60f44025bbdd0d27381" Jul 08 17:47:44.646779 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:44.646735 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2705da7bacdfbd87b76c920a1da8a990102008171cfab60f44025bbdd0d27381"} err="failed to get container status \"2705da7bacdfbd87b76c920a1da8a990102008171cfab60f44025bbdd0d27381\": rpc error: code = NotFound desc = could not find container \"2705da7bacdfbd87b76c920a1da8a990102008171cfab60f44025bbdd0d27381\": container with ID starting with 2705da7bacdfbd87b76c920a1da8a990102008171cfab60f44025bbdd0d27381 not found: ID does not exist" Jul 08 17:47:44.658070 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:44.658030 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-68979f9cc7-wfl5g" podStartSLOduration=2.289169039 podStartE2EDuration="2.658018146s" totalImagesPullingTime="368.849107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:47:44.656002257 +0000 UTC m=+540.340099516" watchObservedRunningTime="2026-07-08 17:47:44.658018146 +0000 UTC m=+540.342115442" Jul 08 17:47:44.672877 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:44.672777 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-jjcc6" podStartSLOduration=1.6355305 podStartE2EDuration="2.672760336s" totalImagesPullingTime="1.037229836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:47:44.670639108 +0000 UTC m=+540.354736385" watchObservedRunningTime="2026-07-08 17:47:44.672760336 +0000 UTC m=+540.356857612" Jul 08 17:47:44.716502 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:44.716451 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-774dfc4d85-kwp77"] Jul 08 17:47:44.722590 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:44.722564 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-774dfc4d85-kwp77"] Jul 08 17:47:44.892644 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:44.892616 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jjcc6" Jul 08 17:47:44.957946 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:44.957868 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75a54d6d-8d0f-404e-af7a-449464111dc5" path="/var/lib/kubelet/pods/75a54d6d-8d0f-404e-af7a-449464111dc5/volumes" Jul 08 17:47:45.011777 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:45.011743 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jf7j6\" (UniqueName: \"kubernetes.io/projected/8fe1c948-c788-45a3-9fd4-f2f86db800a9-kube-api-access-jf7j6\") pod \"8fe1c948-c788-45a3-9fd4-f2f86db800a9\" (UID: \"8fe1c948-c788-45a3-9fd4-f2f86db800a9\") " Jul 08 17:47:45.013921 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:45.013885 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fe1c948-c788-45a3-9fd4-f2f86db800a9-kube-api-access-jf7j6" (OuterVolumeSpecName: "kube-api-access-jf7j6") pod "8fe1c948-c788-45a3-9fd4-f2f86db800a9" (UID: "8fe1c948-c788-45a3-9fd4-f2f86db800a9"). InnerVolumeSpecName "kube-api-access-jf7j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:47:45.112546 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:45.112512 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf7j6\" (UniqueName: \"kubernetes.io/projected/8fe1c948-c788-45a3-9fd4-f2f86db800a9-kube-api-access-jf7j6\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:47:45.645544 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:45.645505 2584 generic.go:356] "Generic (PLEG): container finished" podID="8fe1c948-c788-45a3-9fd4-f2f86db800a9" containerID="06827b981617367547e108621d18677a7617c700450a624acc112103cea46555" exitCode=0 Jul 08 17:47:45.646022 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:45.645548 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jjcc6" Jul 08 17:47:45.646022 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:45.645591 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jjcc6" event={"ID":"8fe1c948-c788-45a3-9fd4-f2f86db800a9","Type":"ContainerDied","Data":"06827b981617367547e108621d18677a7617c700450a624acc112103cea46555"} Jul 08 17:47:45.646022 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:45.645633 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-jjcc6" event={"ID":"8fe1c948-c788-45a3-9fd4-f2f86db800a9","Type":"ContainerDied","Data":"34e48c118159f4f9e24ed7963e2305d2bdc2ebf3f567db4ffe41be20c5bed648"} Jul 08 17:47:45.646022 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:45.645651 2584 scope.go:117] "RemoveContainer" containerID="06827b981617367547e108621d18677a7617c700450a624acc112103cea46555" Jul 08 17:47:45.654447 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:45.654427 2584 scope.go:117] "RemoveContainer" containerID="06827b981617367547e108621d18677a7617c700450a624acc112103cea46555" Jul 08 17:47:45.654717 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:47:45.654699 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06827b981617367547e108621d18677a7617c700450a624acc112103cea46555\": container with ID starting with 06827b981617367547e108621d18677a7617c700450a624acc112103cea46555 not found: ID does not exist" containerID="06827b981617367547e108621d18677a7617c700450a624acc112103cea46555" Jul 08 17:47:45.654778 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:45.654729 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06827b981617367547e108621d18677a7617c700450a624acc112103cea46555"} err="failed to get container status \"06827b981617367547e108621d18677a7617c700450a624acc112103cea46555\": rpc error: code = NotFound desc = could not find container \"06827b981617367547e108621d18677a7617c700450a624acc112103cea46555\": container with ID starting with 06827b981617367547e108621d18677a7617c700450a624acc112103cea46555 not found: ID does not exist" Jul 08 17:47:45.670507 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:45.670468 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jjcc6"] Jul 08 17:47:45.676118 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:45.676086 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jjcc6"] Jul 08 17:47:46.956144 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:46.956109 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fe1c948-c788-45a3-9fd4-f2f86db800a9" path="/var/lib/kubelet/pods/8fe1c948-c788-45a3-9fd4-f2f86db800a9/volumes" Jul 08 17:47:49.061637 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:49.061589 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:49.129520 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:49.129488 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:49.196187 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:49.196149 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:49.270770 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:49.270724 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:51.893265 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:51.893208 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:47:51.993108 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:47:51.993068 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:48:13.764805 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:13.764765 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:48:21.379491 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:21.379445 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:48:22.607140 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:22.607103 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:48:24.947065 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:24.947026 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:48:25.429890 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.429845 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:48:25.504156 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.504123 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z"] Jul 08 17:48:25.504572 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.504547 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a54d6d-8d0f-404e-af7a-449464111dc5" containerName="authorino" Jul 08 17:48:25.504572 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.504567 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a54d6d-8d0f-404e-af7a-449464111dc5" containerName="authorino" Jul 08 17:48:25.504572 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.504579 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fe1c948-c788-45a3-9fd4-f2f86db800a9" containerName="authorino" Jul 08 17:48:25.504895 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.504585 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fe1c948-c788-45a3-9fd4-f2f86db800a9" containerName="authorino" Jul 08 17:48:25.504895 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.504659 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="75a54d6d-8d0f-404e-af7a-449464111dc5" containerName="authorino" Jul 08 17:48:25.504895 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.504670 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fe1c948-c788-45a3-9fd4-f2f86db800a9" containerName="authorino" Jul 08 17:48:25.507984 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.507964 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z" Jul 08 17:48:25.510309 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.510284 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 08 17:48:25.510419 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.510328 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mzmms\"" Jul 08 17:48:25.519283 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.519260 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z"] Jul 08 17:48:25.570051 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.570009 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5zkd\" (UniqueName: \"kubernetes.io/projected/7573357e-6f6f-48b5-be9c-640f678eb06d-kube-api-access-c5zkd\") pod \"maas-api-5c4dddb657-nqg4z\" (UID: \"7573357e-6f6f-48b5-be9c-640f678eb06d\") " pod="odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z" Jul 08 17:48:25.570051 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.570050 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7573357e-6f6f-48b5-be9c-640f678eb06d-openshift-service-ca\") pod \"maas-api-5c4dddb657-nqg4z\" (UID: \"7573357e-6f6f-48b5-be9c-640f678eb06d\") " pod="odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z" Jul 08 17:48:25.570311 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.570144 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7573357e-6f6f-48b5-be9c-640f678eb06d-maas-api-tls\") pod \"maas-api-5c4dddb657-nqg4z\" (UID: \"7573357e-6f6f-48b5-be9c-640f678eb06d\") " pod="odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z" Jul 08 17:48:25.671280 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.671197 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5zkd\" (UniqueName: \"kubernetes.io/projected/7573357e-6f6f-48b5-be9c-640f678eb06d-kube-api-access-c5zkd\") pod \"maas-api-5c4dddb657-nqg4z\" (UID: \"7573357e-6f6f-48b5-be9c-640f678eb06d\") " pod="odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z" Jul 08 17:48:25.671486 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.671292 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7573357e-6f6f-48b5-be9c-640f678eb06d-openshift-service-ca\") pod \"maas-api-5c4dddb657-nqg4z\" (UID: \"7573357e-6f6f-48b5-be9c-640f678eb06d\") " pod="odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z" Jul 08 17:48:25.671486 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.671371 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7573357e-6f6f-48b5-be9c-640f678eb06d-maas-api-tls\") pod \"maas-api-5c4dddb657-nqg4z\" (UID: \"7573357e-6f6f-48b5-be9c-640f678eb06d\") " pod="odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z" Jul 08 17:48:25.671623 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:48:25.671485 2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 08 17:48:25.671623 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:48:25.671557 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7573357e-6f6f-48b5-be9c-640f678eb06d-maas-api-tls podName:7573357e-6f6f-48b5-be9c-640f678eb06d nodeName:}" failed. No retries permitted until 2026-07-08 17:48:26.171535452 +0000 UTC m=+581.855632721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7573357e-6f6f-48b5-be9c-640f678eb06d-maas-api-tls") pod "maas-api-5c4dddb657-nqg4z" (UID: "7573357e-6f6f-48b5-be9c-640f678eb06d") : secret "maas-api-serving-cert" not found Jul 08 17:48:25.671930 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.671908 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7573357e-6f6f-48b5-be9c-640f678eb06d-openshift-service-ca\") pod \"maas-api-5c4dddb657-nqg4z\" (UID: \"7573357e-6f6f-48b5-be9c-640f678eb06d\") " pod="odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z" Jul 08 17:48:25.681638 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.681572 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5zkd\" (UniqueName: \"kubernetes.io/projected/7573357e-6f6f-48b5-be9c-640f678eb06d-kube-api-access-c5zkd\") pod \"maas-api-5c4dddb657-nqg4z\" (UID: \"7573357e-6f6f-48b5-be9c-640f678eb06d\") " pod="odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z" Jul 08 17:48:25.907919 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:25.907881 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:48:26.008074 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:26.007977 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:48:26.176263 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:26.176201 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7573357e-6f6f-48b5-be9c-640f678eb06d-maas-api-tls\") pod \"maas-api-5c4dddb657-nqg4z\" (UID: \"7573357e-6f6f-48b5-be9c-640f678eb06d\") " pod="odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z" Jul 08 17:48:26.179001 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:26.178974 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7573357e-6f6f-48b5-be9c-640f678eb06d-maas-api-tls\") pod \"maas-api-5c4dddb657-nqg4z\" (UID: \"7573357e-6f6f-48b5-be9c-640f678eb06d\") " pod="odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z" Jul 08 17:48:26.419617 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:26.419566 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z" Jul 08 17:48:26.568665 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:26.568636 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z"] Jul 08 17:48:26.570931 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:48:26.570893 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7573357e_6f6f_48b5_be9c_640f678eb06d.slice/crio-85492de9acce365f888e4203f545407087cbd1246387e51e2a392143e6581bbe WatchSource:0}: Error finding container 85492de9acce365f888e4203f545407087cbd1246387e51e2a392143e6581bbe: Status 404 returned error can't find the container with id 85492de9acce365f888e4203f545407087cbd1246387e51e2a392143e6581bbe Jul 08 17:48:26.869840 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:26.869805 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z" event={"ID":"7573357e-6f6f-48b5-be9c-640f678eb06d","Type":"ContainerStarted","Data":"85492de9acce365f888e4203f545407087cbd1246387e51e2a392143e6581bbe"} Jul 08 17:48:27.279226 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:27.279140 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:48:28.883061 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:28.883026 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z" event={"ID":"7573357e-6f6f-48b5-be9c-640f678eb06d","Type":"ContainerStarted","Data":"9d0195dfdb78d413aab593e4be786704cf643a6a18e37779554e52d1e0b3f233"} Jul 08 17:48:28.883520 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:28.883165 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z" Jul 08 17:48:28.917067 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:28.917019 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z" podStartSLOduration=2.040867379 podStartE2EDuration="3.917004379s" totalImagesPullingTime="1.876137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:48:28.914384927 +0000 UTC m=+584.598482205" watchObservedRunningTime="2026-07-08 17:48:28.917004379 +0000 UTC m=+584.601101656" Jul 08 17:48:34.618177 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:34.618145 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:48:34.719475 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:34.719442 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:48:34.894461 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:34.894378 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c4dddb657-nqg4z" Jul 08 17:48:35.879922 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:35.879878 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:48:35.980373 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:35.980327 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:48:44.801871 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:44.801841 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/1.log" Jul 08 17:48:44.803063 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:44.803040 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/1.log" Jul 08 17:48:50.289147 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.289107 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl"] Jul 08 17:48:50.294050 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.294029 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.297925 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.297902 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 08 17:48:50.298085 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.297962 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 08 17:48:50.298085 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.297902 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-w5cfc\"" Jul 08 17:48:50.298085 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.297902 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 08 17:48:50.303886 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.303859 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl"] Jul 08 17:48:50.336031 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.335993 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:48:50.390352 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.390315 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f028844-9c43-410b-ba1b-cad906fb0553-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl\" (UID: \"2f028844-9c43-410b-ba1b-cad906fb0553\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.390352 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.390359 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f028844-9c43-410b-ba1b-cad906fb0553-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl\" (UID: \"2f028844-9c43-410b-ba1b-cad906fb0553\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.390589 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.390431 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f028844-9c43-410b-ba1b-cad906fb0553-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl\" (UID: \"2f028844-9c43-410b-ba1b-cad906fb0553\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.390589 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.390461 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f028844-9c43-410b-ba1b-cad906fb0553-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl\" (UID: \"2f028844-9c43-410b-ba1b-cad906fb0553\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.390589 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.390507 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f028844-9c43-410b-ba1b-cad906fb0553-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl\" (UID: \"2f028844-9c43-410b-ba1b-cad906fb0553\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.390589 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.390576 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p98h\" (UniqueName: \"kubernetes.io/projected/2f028844-9c43-410b-ba1b-cad906fb0553-kube-api-access-7p98h\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl\" (UID: \"2f028844-9c43-410b-ba1b-cad906fb0553\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.492048 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.492007 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f028844-9c43-410b-ba1b-cad906fb0553-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl\" (UID: \"2f028844-9c43-410b-ba1b-cad906fb0553\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.492220 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.492066 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p98h\" (UniqueName: \"kubernetes.io/projected/2f028844-9c43-410b-ba1b-cad906fb0553-kube-api-access-7p98h\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl\" (UID: \"2f028844-9c43-410b-ba1b-cad906fb0553\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.492220 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.492101 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f028844-9c43-410b-ba1b-cad906fb0553-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl\" (UID: \"2f028844-9c43-410b-ba1b-cad906fb0553\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.492220 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.492121 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f028844-9c43-410b-ba1b-cad906fb0553-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl\" (UID: \"2f028844-9c43-410b-ba1b-cad906fb0553\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.492220 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.492141 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f028844-9c43-410b-ba1b-cad906fb0553-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl\" (UID: \"2f028844-9c43-410b-ba1b-cad906fb0553\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.492220 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.492162 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f028844-9c43-410b-ba1b-cad906fb0553-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl\" (UID: \"2f028844-9c43-410b-ba1b-cad906fb0553\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.492527 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.492501 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f028844-9c43-410b-ba1b-cad906fb0553-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl\" (UID: \"2f028844-9c43-410b-ba1b-cad906fb0553\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.492628 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.492583 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f028844-9c43-410b-ba1b-cad906fb0553-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl\" (UID: \"2f028844-9c43-410b-ba1b-cad906fb0553\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.492749 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.492627 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f028844-9c43-410b-ba1b-cad906fb0553-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl\" (UID: \"2f028844-9c43-410b-ba1b-cad906fb0553\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.494395 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.494375 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f028844-9c43-410b-ba1b-cad906fb0553-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl\" (UID: \"2f028844-9c43-410b-ba1b-cad906fb0553\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.494672 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.494653 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f028844-9c43-410b-ba1b-cad906fb0553-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl\" (UID: \"2f028844-9c43-410b-ba1b-cad906fb0553\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.503137 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.503113 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p98h\" (UniqueName: \"kubernetes.io/projected/2f028844-9c43-410b-ba1b-cad906fb0553-kube-api-access-7p98h\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl\" (UID: \"2f028844-9c43-410b-ba1b-cad906fb0553\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.606547 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.606458 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:48:50.756681 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:50.756655 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl"] Jul 08 17:48:50.758009 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:48:50.757981 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f028844_9c43_410b_ba1b_cad906fb0553.slice/crio-10016df6dab8e83ff68a7fd675a3a9608c454212185e4b52cce92bb9045bb036 WatchSource:0}: Error finding container 10016df6dab8e83ff68a7fd675a3a9608c454212185e4b52cce92bb9045bb036: Status 404 returned error can't find the container with id 10016df6dab8e83ff68a7fd675a3a9608c454212185e4b52cce92bb9045bb036 Jul 08 17:48:51.013547 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:51.013435 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" event={"ID":"2f028844-9c43-410b-ba1b-cad906fb0553","Type":"ContainerStarted","Data":"10016df6dab8e83ff68a7fd675a3a9608c454212185e4b52cce92bb9045bb036"} Jul 08 17:48:52.021290 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:52.021250 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" event={"ID":"2f028844-9c43-410b-ba1b-cad906fb0553","Type":"ContainerStarted","Data":"90f58976803fcdc7553f911bdb7a53b043f1c040aa4f448ad6da5f41818fd0a7"} Jul 08 17:48:52.042548 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:52.042498 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" podStartSLOduration=0.886194192 podStartE2EDuration="2.042479057s" totalImagesPullingTime="1.156284865s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:48:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:48:52.040738191 +0000 UTC m=+607.724835471" watchObservedRunningTime="2026-07-08 17:48:52.042479057 +0000 UTC m=+607.726576333" Jul 08 17:48:56.912162 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:48:56.912125 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:49:00.607196 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:49:00.607155 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:49:00.607622 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:49:00.607229 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:49:00.619977 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:49:00.619957 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:49:01.082080 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:49:01.082052 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl" Jul 08 17:49:14.019790 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:49:14.019710 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:49:30.772906 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:49:30.772863 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:49:34.894951 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:49:34.894911 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:49:59.896117 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:49:59.896075 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:50:04.356551 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:50:04.356505 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:50:04.413619 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:50:04.413575 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:50:10.600986 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:50:10.600947 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:50:23.897118 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:50:23.897081 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:51:20.807563 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:20.807530 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c44cb9c6-62sng"] Jul 08 17:51:20.811343 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:20.811319 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c44cb9c6-62sng" Jul 08 17:51:20.818319 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:20.818292 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c44cb9c6-62sng"] Jul 08 17:51:20.901983 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:20.901935 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld27f\" (UniqueName: \"kubernetes.io/projected/577df30a-813c-4063-bc7f-3c9d83edea32-kube-api-access-ld27f\") pod \"authorino-5c44cb9c6-62sng\" (UID: \"577df30a-813c-4063-bc7f-3c9d83edea32\") " pod="kuadrant-system/authorino-5c44cb9c6-62sng" Jul 08 17:51:20.901983 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:20.901977 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/577df30a-813c-4063-bc7f-3c9d83edea32-tls-cert\") pod \"authorino-5c44cb9c6-62sng\" (UID: \"577df30a-813c-4063-bc7f-3c9d83edea32\") " pod="kuadrant-system/authorino-5c44cb9c6-62sng" Jul 08 17:51:21.003113 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:21.003074 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld27f\" (UniqueName: \"kubernetes.io/projected/577df30a-813c-4063-bc7f-3c9d83edea32-kube-api-access-ld27f\") pod \"authorino-5c44cb9c6-62sng\" (UID: \"577df30a-813c-4063-bc7f-3c9d83edea32\") " pod="kuadrant-system/authorino-5c44cb9c6-62sng" Jul 08 17:51:21.003113 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:21.003114 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/577df30a-813c-4063-bc7f-3c9d83edea32-tls-cert\") pod \"authorino-5c44cb9c6-62sng\" (UID: \"577df30a-813c-4063-bc7f-3c9d83edea32\") " pod="kuadrant-system/authorino-5c44cb9c6-62sng" Jul 08 17:51:21.005559 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:21.005537 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/577df30a-813c-4063-bc7f-3c9d83edea32-tls-cert\") pod \"authorino-5c44cb9c6-62sng\" (UID: \"577df30a-813c-4063-bc7f-3c9d83edea32\") " pod="kuadrant-system/authorino-5c44cb9c6-62sng" Jul 08 17:51:21.013076 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:21.013054 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld27f\" (UniqueName: \"kubernetes.io/projected/577df30a-813c-4063-bc7f-3c9d83edea32-kube-api-access-ld27f\") pod \"authorino-5c44cb9c6-62sng\" (UID: \"577df30a-813c-4063-bc7f-3c9d83edea32\") " pod="kuadrant-system/authorino-5c44cb9c6-62sng" Jul 08 17:51:21.122654 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:21.122624 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c44cb9c6-62sng" Jul 08 17:51:21.211875 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:21.211846 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:51:21.269553 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:21.269527 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c44cb9c6-62sng"] Jul 08 17:51:21.271066 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:51:21.271040 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod577df30a_813c_4063_bc7f_3c9d83edea32.slice/crio-7f9e1033be851b3090a105f9eaf49b44829adce8710f60b056b56698702bdfb2 WatchSource:0}: Error finding container 7f9e1033be851b3090a105f9eaf49b44829adce8710f60b056b56698702bdfb2: Status 404 returned error can't find the container with id 7f9e1033be851b3090a105f9eaf49b44829adce8710f60b056b56698702bdfb2 Jul 08 17:51:21.272562 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:21.272542 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 17:51:21.275283 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:21.275255 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:51:21.861390 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:21.861292 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c44cb9c6-62sng" event={"ID":"577df30a-813c-4063-bc7f-3c9d83edea32","Type":"ContainerStarted","Data":"e5de03df8b26467be3d1179befa572ad09eca8106b20030da1b20acff3798dc6"} Jul 08 17:51:21.861390 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:21.861343 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c44cb9c6-62sng" event={"ID":"577df30a-813c-4063-bc7f-3c9d83edea32","Type":"ContainerStarted","Data":"7f9e1033be851b3090a105f9eaf49b44829adce8710f60b056b56698702bdfb2"} Jul 08 17:51:21.893045 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:21.892980 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5c44cb9c6-62sng" podStartSLOduration=1.5610556340000001 podStartE2EDuration="1.892956341s" totalImagesPullingTime="331.900707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:51:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:51:21.8916177 +0000 UTC m=+757.575714999" watchObservedRunningTime="2026-07-08 17:51:21.892956341 +0000 UTC m=+757.577053618" Jul 08 17:51:21.940374 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:21.939601 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68979f9cc7-wfl5g"] Jul 08 17:51:21.940374 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:21.939814 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-68979f9cc7-wfl5g" podUID="e1775e1d-763f-482a-b327-da8ad008d9be" containerName="authorino" containerID="cri-o://162ad1e65ef769be12f0a011c4c3de02bb2acc347f9fb301ef0a3c40cb1422eb" gracePeriod=30 Jul 08 17:51:22.199104 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:22.199069 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68979f9cc7-wfl5g" Jul 08 17:51:22.212615 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:22.212584 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vckf2\" (UniqueName: \"kubernetes.io/projected/e1775e1d-763f-482a-b327-da8ad008d9be-kube-api-access-vckf2\") pod \"e1775e1d-763f-482a-b327-da8ad008d9be\" (UID: \"e1775e1d-763f-482a-b327-da8ad008d9be\") " Jul 08 17:51:22.212766 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:22.212637 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e1775e1d-763f-482a-b327-da8ad008d9be-tls-cert\") pod \"e1775e1d-763f-482a-b327-da8ad008d9be\" (UID: \"e1775e1d-763f-482a-b327-da8ad008d9be\") " Jul 08 17:51:22.214791 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:22.214755 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1775e1d-763f-482a-b327-da8ad008d9be-kube-api-access-vckf2" (OuterVolumeSpecName: "kube-api-access-vckf2") pod "e1775e1d-763f-482a-b327-da8ad008d9be" (UID: "e1775e1d-763f-482a-b327-da8ad008d9be"). InnerVolumeSpecName "kube-api-access-vckf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:51:22.227327 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:22.227291 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1775e1d-763f-482a-b327-da8ad008d9be-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e1775e1d-763f-482a-b327-da8ad008d9be" (UID: "e1775e1d-763f-482a-b327-da8ad008d9be"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 17:51:22.313916 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:22.313879 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vckf2\" (UniqueName: \"kubernetes.io/projected/e1775e1d-763f-482a-b327-da8ad008d9be-kube-api-access-vckf2\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:51:22.313916 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:22.313919 2584 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e1775e1d-763f-482a-b327-da8ad008d9be-tls-cert\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:51:22.406123 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:22.405972 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:51:22.507498 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:22.507459 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:51:22.868871 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:22.868843 2584 generic.go:356] "Generic (PLEG): container finished" podID="e1775e1d-763f-482a-b327-da8ad008d9be" containerID="162ad1e65ef769be12f0a011c4c3de02bb2acc347f9fb301ef0a3c40cb1422eb" exitCode=0 Jul 08 17:51:22.869357 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:22.868888 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68979f9cc7-wfl5g" Jul 08 17:51:22.869357 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:22.868933 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68979f9cc7-wfl5g" event={"ID":"e1775e1d-763f-482a-b327-da8ad008d9be","Type":"ContainerDied","Data":"162ad1e65ef769be12f0a011c4c3de02bb2acc347f9fb301ef0a3c40cb1422eb"} Jul 08 17:51:22.869357 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:22.868977 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68979f9cc7-wfl5g" event={"ID":"e1775e1d-763f-482a-b327-da8ad008d9be","Type":"ContainerDied","Data":"b5467df89163757fdd323daa13619f3d1795c2a19405343e01a3b350f9dde2c5"} Jul 08 17:51:22.869357 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:22.869004 2584 scope.go:117] "RemoveContainer" containerID="162ad1e65ef769be12f0a011c4c3de02bb2acc347f9fb301ef0a3c40cb1422eb" Jul 08 17:51:22.878633 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:22.878605 2584 scope.go:117] "RemoveContainer" containerID="162ad1e65ef769be12f0a011c4c3de02bb2acc347f9fb301ef0a3c40cb1422eb" Jul 08 17:51:22.878876 ip-10-0-129-226 kubenswrapper[2584]: E0708 17:51:22.878858 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"162ad1e65ef769be12f0a011c4c3de02bb2acc347f9fb301ef0a3c40cb1422eb\": container with ID starting with 162ad1e65ef769be12f0a011c4c3de02bb2acc347f9fb301ef0a3c40cb1422eb not found: ID does not exist" containerID="162ad1e65ef769be12f0a011c4c3de02bb2acc347f9fb301ef0a3c40cb1422eb" Jul 08 17:51:22.878936 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:22.878883 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"162ad1e65ef769be12f0a011c4c3de02bb2acc347f9fb301ef0a3c40cb1422eb"} err="failed to get container status \"162ad1e65ef769be12f0a011c4c3de02bb2acc347f9fb301ef0a3c40cb1422eb\": rpc error: code = NotFound desc = could not find container \"162ad1e65ef769be12f0a011c4c3de02bb2acc347f9fb301ef0a3c40cb1422eb\": container with ID starting with 162ad1e65ef769be12f0a011c4c3de02bb2acc347f9fb301ef0a3c40cb1422eb not found: ID does not exist" Jul 08 17:51:22.895715 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:22.895686 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-68979f9cc7-wfl5g"] Jul 08 17:51:22.898151 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:22.898127 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-68979f9cc7-wfl5g"] Jul 08 17:51:22.957897 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:51:22.957863 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1775e1d-763f-482a-b327-da8ad008d9be" path="/var/lib/kubelet/pods/e1775e1d-763f-482a-b327-da8ad008d9be/volumes" Jul 08 17:52:33.287351 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:52:33.287230 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:52:34.615498 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:52:34.615462 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:53:14.960092 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:53:14.960049 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:53:15.059182 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:53:15.059144 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:53:15.173433 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:53:15.173395 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:53:21.520037 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:53:21.520002 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:53:21.625184 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:53:21.625148 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:53:21.722198 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:53:21.722162 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:53:21.822225 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:53:21.822141 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:53:28.037167 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:53:28.037102 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:53:28.256746 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:53:28.256706 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:53:44.848727 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:53:44.848688 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/1.log" Jul 08 17:53:44.850711 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:53:44.850687 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/1.log" Jul 08 17:54:16.622265 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:54:16.622211 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:54:50.285686 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:54:50.285648 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:57:26.240326 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:57:26.240294 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz"] Jul 08 17:57:26.240876 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:57:26.240547 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz" podUID="b27da8f8-ac5e-4c39-ae38-10575f4548a1" containerName="manager" containerID="cri-o://75ba9ab7fd060a5e0577d2522926b847ed3c09381285d38172c1421c2864361e" gracePeriod=10 Jul 08 17:57:29.971823 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:57:29.971786 2584 generic.go:356] "Generic (PLEG): container finished" podID="b27da8f8-ac5e-4c39-ae38-10575f4548a1" containerID="75ba9ab7fd060a5e0577d2522926b847ed3c09381285d38172c1421c2864361e" exitCode=0 Jul 08 17:57:29.972184 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:57:29.971837 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz" event={"ID":"b27da8f8-ac5e-4c39-ae38-10575f4548a1","Type":"ContainerDied","Data":"75ba9ab7fd060a5e0577d2522926b847ed3c09381285d38172c1421c2864361e"} Jul 08 17:57:30.497620 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:57:30.497593 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz" Jul 08 17:57:30.594909 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:57:30.594874 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b27da8f8-ac5e-4c39-ae38-10575f4548a1-extensions-socket-volume\") pod \"b27da8f8-ac5e-4c39-ae38-10575f4548a1\" (UID: \"b27da8f8-ac5e-4c39-ae38-10575f4548a1\") " Jul 08 17:57:30.595105 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:57:30.594963 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxbhc\" (UniqueName: \"kubernetes.io/projected/b27da8f8-ac5e-4c39-ae38-10575f4548a1-kube-api-access-lxbhc\") pod \"b27da8f8-ac5e-4c39-ae38-10575f4548a1\" (UID: \"b27da8f8-ac5e-4c39-ae38-10575f4548a1\") " Jul 08 17:57:30.595357 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:57:30.595326 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b27da8f8-ac5e-4c39-ae38-10575f4548a1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b27da8f8-ac5e-4c39-ae38-10575f4548a1" (UID: "b27da8f8-ac5e-4c39-ae38-10575f4548a1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 17:57:30.596940 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:57:30.596915 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27da8f8-ac5e-4c39-ae38-10575f4548a1-kube-api-access-lxbhc" (OuterVolumeSpecName: "kube-api-access-lxbhc") pod "b27da8f8-ac5e-4c39-ae38-10575f4548a1" (UID: "b27da8f8-ac5e-4c39-ae38-10575f4548a1"). InnerVolumeSpecName "kube-api-access-lxbhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 17:57:30.696370 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:57:30.696330 2584 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b27da8f8-ac5e-4c39-ae38-10575f4548a1-extensions-socket-volume\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:57:30.696370 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:57:30.696362 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxbhc\" (UniqueName: \"kubernetes.io/projected/b27da8f8-ac5e-4c39-ae38-10575f4548a1-kube-api-access-lxbhc\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 17:57:30.978079 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:57:30.978045 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz" event={"ID":"b27da8f8-ac5e-4c39-ae38-10575f4548a1","Type":"ContainerDied","Data":"bdd7e543c1566aa686250257e064ded2e2a22643647b46fa590e8dc3d421a021"} Jul 08 17:57:30.978079 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:57:30.978073 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz" Jul 08 17:57:30.978532 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:57:30.978119 2584 scope.go:117] "RemoveContainer" containerID="75ba9ab7fd060a5e0577d2522926b847ed3c09381285d38172c1421c2864361e" Jul 08 17:57:31.006313 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:57:31.006283 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz"] Jul 08 17:57:31.021696 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:57:31.021662 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-mjqbz"] Jul 08 17:57:32.956564 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:57:32.956528 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b27da8f8-ac5e-4c39-ae38-10575f4548a1" path="/var/lib/kubelet/pods/b27da8f8-ac5e-4c39-ae38-10575f4548a1/volumes" Jul 08 17:57:42.219625 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:57:42.219584 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:57:42.280009 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:57:42.279967 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:58:02.148386 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:02.148345 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:58:02.202381 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:02.202348 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:58:27.343895 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:27.343819 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7jdjn"] Jul 08 17:58:27.344364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:27.344273 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1775e1d-763f-482a-b327-da8ad008d9be" containerName="authorino" Jul 08 17:58:27.344364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:27.344287 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1775e1d-763f-482a-b327-da8ad008d9be" containerName="authorino" Jul 08 17:58:27.344364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:27.344296 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b27da8f8-ac5e-4c39-ae38-10575f4548a1" containerName="manager" Jul 08 17:58:27.344364 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:27.344301 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="b27da8f8-ac5e-4c39-ae38-10575f4548a1" containerName="manager" Jul 08 17:58:27.344513 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:27.344381 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1775e1d-763f-482a-b327-da8ad008d9be" containerName="authorino" Jul 08 17:58:27.344513 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:27.344391 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="b27da8f8-ac5e-4c39-ae38-10575f4548a1" containerName="manager" Jul 08 17:58:27.347502 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:27.347480 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7jdjn" Jul 08 17:58:27.349910 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:27.349891 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kqmbf\"" Jul 08 17:58:27.363017 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:27.362990 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7jdjn"] Jul 08 17:58:27.409480 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:27.409438 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a936399-3688-4348-ad1d-28c3c02060f6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7jdjn\" (UID: \"8a936399-3688-4348-ad1d-28c3c02060f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7jdjn" Jul 08 17:58:27.409480 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:27.409481 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7qd2\" (UniqueName: \"kubernetes.io/projected/8a936399-3688-4348-ad1d-28c3c02060f6-kube-api-access-v7qd2\") pod \"kuadrant-operator-controller-manager-86878c769f-7jdjn\" (UID: \"8a936399-3688-4348-ad1d-28c3c02060f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7jdjn" Jul 08 17:58:27.510869 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:27.510829 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a936399-3688-4348-ad1d-28c3c02060f6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7jdjn\" (UID: \"8a936399-3688-4348-ad1d-28c3c02060f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7jdjn" Jul 08 17:58:27.510869 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:27.510872 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7qd2\" (UniqueName: \"kubernetes.io/projected/8a936399-3688-4348-ad1d-28c3c02060f6-kube-api-access-v7qd2\") pod \"kuadrant-operator-controller-manager-86878c769f-7jdjn\" (UID: \"8a936399-3688-4348-ad1d-28c3c02060f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7jdjn" Jul 08 17:58:27.511372 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:27.511350 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a936399-3688-4348-ad1d-28c3c02060f6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7jdjn\" (UID: \"8a936399-3688-4348-ad1d-28c3c02060f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7jdjn" Jul 08 17:58:27.520332 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:27.520306 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7qd2\" (UniqueName: \"kubernetes.io/projected/8a936399-3688-4348-ad1d-28c3c02060f6-kube-api-access-v7qd2\") pod \"kuadrant-operator-controller-manager-86878c769f-7jdjn\" (UID: \"8a936399-3688-4348-ad1d-28c3c02060f6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7jdjn" Jul 08 17:58:27.660518 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:27.660472 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7jdjn" Jul 08 17:58:27.803314 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:27.803286 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7jdjn"] Jul 08 17:58:27.805979 ip-10-0-129-226 kubenswrapper[2584]: W0708 17:58:27.805952 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a936399_3688_4348_ad1d_28c3c02060f6.slice/crio-bd2289aa77d487ac615012a28db8b45b39d7a8ea456e986b666fbbadbb870f97 WatchSource:0}: Error finding container bd2289aa77d487ac615012a28db8b45b39d7a8ea456e986b666fbbadbb870f97: Status 404 returned error can't find the container with id bd2289aa77d487ac615012a28db8b45b39d7a8ea456e986b666fbbadbb870f97 Jul 08 17:58:28.292679 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:28.292643 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7jdjn" event={"ID":"8a936399-3688-4348-ad1d-28c3c02060f6","Type":"ContainerStarted","Data":"01ba58c788de0e30aa1ef4818e9c79a75905a49acd7a90802f076b25de387a37"} Jul 08 17:58:28.292844 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:28.292700 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7jdjn" event={"ID":"8a936399-3688-4348-ad1d-28c3c02060f6","Type":"ContainerStarted","Data":"bd2289aa77d487ac615012a28db8b45b39d7a8ea456e986b666fbbadbb870f97"} Jul 08 17:58:28.292844 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:28.292773 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7jdjn" Jul 08 17:58:28.326010 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:28.325955 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7jdjn" podStartSLOduration=1.325939408 podStartE2EDuration="1.325939408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:58:28.322873329 +0000 UTC m=+1184.006970617" watchObservedRunningTime="2026-07-08 17:58:28.325939408 +0000 UTC m=+1184.010036783" Jul 08 17:58:39.300216 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:39.300188 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7jdjn" Jul 08 17:58:44.896972 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:44.896941 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/1.log" Jul 08 17:58:44.897640 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:44.897619 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/1.log" Jul 08 17:58:48.281830 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:48.281785 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 17:58:59.767686 ip-10-0-129-226 kubenswrapper[2584]: I0708 17:58:59.767646 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:00:00.154551 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:00:00.154512 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725560-5c4gd"] Jul 08 18:00:00.158104 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:00:00.158084 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725560-5c4gd" Jul 08 18:00:00.165388 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:00:00.165356 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725560-5c4gd"] Jul 08 18:00:00.187873 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:00:00.187824 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkzm8\" (UniqueName: \"kubernetes.io/projected/4c7ac194-f1de-4fbc-bdd9-85501139b868-kube-api-access-xkzm8\") pod \"maas-api-key-cleanup-29725560-5c4gd\" (UID: \"4c7ac194-f1de-4fbc-bdd9-85501139b868\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725560-5c4gd" Jul 08 18:00:00.289349 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:00:00.289307 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkzm8\" (UniqueName: \"kubernetes.io/projected/4c7ac194-f1de-4fbc-bdd9-85501139b868-kube-api-access-xkzm8\") pod \"maas-api-key-cleanup-29725560-5c4gd\" (UID: \"4c7ac194-f1de-4fbc-bdd9-85501139b868\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725560-5c4gd" Jul 08 18:00:00.300624 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:00:00.300593 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkzm8\" (UniqueName: \"kubernetes.io/projected/4c7ac194-f1de-4fbc-bdd9-85501139b868-kube-api-access-xkzm8\") pod \"maas-api-key-cleanup-29725560-5c4gd\" (UID: \"4c7ac194-f1de-4fbc-bdd9-85501139b868\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725560-5c4gd" Jul 08 18:00:00.469963 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:00:00.469874 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725560-5c4gd" Jul 08 18:00:00.610741 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:00:00.610705 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725560-5c4gd"] Jul 08 18:00:00.612032 ip-10-0-129-226 kubenswrapper[2584]: W0708 18:00:00.611995 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c7ac194_f1de_4fbc_bdd9_85501139b868.slice/crio-bf00719476115dff6f228a80d5a9ee199c0f0fb8673f18b9025d891c2eaa2220 WatchSource:0}: Error finding container bf00719476115dff6f228a80d5a9ee199c0f0fb8673f18b9025d891c2eaa2220: Status 404 returned error can't find the container with id bf00719476115dff6f228a80d5a9ee199c0f0fb8673f18b9025d891c2eaa2220 Jul 08 18:00:00.614458 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:00:00.614437 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 18:00:00.817147 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:00:00.817055 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725560-5c4gd" event={"ID":"4c7ac194-f1de-4fbc-bdd9-85501139b868","Type":"ContainerStarted","Data":"bf00719476115dff6f228a80d5a9ee199c0f0fb8673f18b9025d891c2eaa2220"} Jul 08 18:00:04.843321 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:00:04.843277 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725560-5c4gd" event={"ID":"4c7ac194-f1de-4fbc-bdd9-85501139b868","Type":"ContainerStarted","Data":"6572e271617e0cdf69974c8725eea54c2e8a3e1d4070ef61b339267f7c665c58"} Jul 08 18:00:04.862046 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:00:04.861988 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725560-5c4gd" podStartSLOduration=1.492947674 podStartE2EDuration="4.861971015s" totalImagesPullingTime="3.369023341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:00:04.860532201 +0000 UTC m=+1280.544629490" watchObservedRunningTime="2026-07-08 18:00:04.861971015 +0000 UTC m=+1280.546068292" Jul 08 18:01:00.011573 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:01:00.011534 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725560-5c4gd"] Jul 08 18:01:00.012100 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:01:00.011738 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725560-5c4gd" podUID="4c7ac194-f1de-4fbc-bdd9-85501139b868" containerName="cleanup" containerID="cri-o://6572e271617e0cdf69974c8725eea54c2e8a3e1d4070ef61b339267f7c665c58" gracePeriod=30 Jul 08 18:01:20.530971 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:01:20.530891 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:01:30.199937 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:01:30.199905 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725560-5c4gd" Jul 08 18:01:30.324779 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:01:30.324681 2584 generic.go:356] "Generic (PLEG): container finished" podID="4c7ac194-f1de-4fbc-bdd9-85501139b868" containerID="6572e271617e0cdf69974c8725eea54c2e8a3e1d4070ef61b339267f7c665c58" exitCode=137 Jul 08 18:01:30.324779 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:01:30.324724 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725560-5c4gd" Jul 08 18:01:30.324779 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:01:30.324752 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725560-5c4gd" event={"ID":"4c7ac194-f1de-4fbc-bdd9-85501139b868","Type":"ContainerDied","Data":"6572e271617e0cdf69974c8725eea54c2e8a3e1d4070ef61b339267f7c665c58"} Jul 08 18:01:30.325022 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:01:30.324805 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725560-5c4gd" event={"ID":"4c7ac194-f1de-4fbc-bdd9-85501139b868","Type":"ContainerDied","Data":"bf00719476115dff6f228a80d5a9ee199c0f0fb8673f18b9025d891c2eaa2220"} Jul 08 18:01:30.325022 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:01:30.324834 2584 scope.go:117] "RemoveContainer" containerID="6572e271617e0cdf69974c8725eea54c2e8a3e1d4070ef61b339267f7c665c58" Jul 08 18:01:30.335117 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:01:30.335095 2584 scope.go:117] "RemoveContainer" containerID="6572e271617e0cdf69974c8725eea54c2e8a3e1d4070ef61b339267f7c665c58" Jul 08 18:01:30.335402 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:01:30.335383 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6572e271617e0cdf69974c8725eea54c2e8a3e1d4070ef61b339267f7c665c58\": container with ID starting with 6572e271617e0cdf69974c8725eea54c2e8a3e1d4070ef61b339267f7c665c58 not found: ID does not exist" containerID="6572e271617e0cdf69974c8725eea54c2e8a3e1d4070ef61b339267f7c665c58" Jul 08 18:01:30.335480 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:01:30.335409 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6572e271617e0cdf69974c8725eea54c2e8a3e1d4070ef61b339267f7c665c58"} err="failed to get container status \"6572e271617e0cdf69974c8725eea54c2e8a3e1d4070ef61b339267f7c665c58\": rpc error: code = NotFound desc = could not find container \"6572e271617e0cdf69974c8725eea54c2e8a3e1d4070ef61b339267f7c665c58\": container with ID starting with 6572e271617e0cdf69974c8725eea54c2e8a3e1d4070ef61b339267f7c665c58 not found: ID does not exist" Jul 08 18:01:30.371492 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:01:30.371457 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkzm8\" (UniqueName: \"kubernetes.io/projected/4c7ac194-f1de-4fbc-bdd9-85501139b868-kube-api-access-xkzm8\") pod \"4c7ac194-f1de-4fbc-bdd9-85501139b868\" (UID: \"4c7ac194-f1de-4fbc-bdd9-85501139b868\") " Jul 08 18:01:30.373638 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:01:30.373605 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c7ac194-f1de-4fbc-bdd9-85501139b868-kube-api-access-xkzm8" (OuterVolumeSpecName: "kube-api-access-xkzm8") pod "4c7ac194-f1de-4fbc-bdd9-85501139b868" (UID: "4c7ac194-f1de-4fbc-bdd9-85501139b868"). InnerVolumeSpecName "kube-api-access-xkzm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:01:30.472100 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:01:30.472066 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkzm8\" (UniqueName: \"kubernetes.io/projected/4c7ac194-f1de-4fbc-bdd9-85501139b868-kube-api-access-xkzm8\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:01:30.648286 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:01:30.648234 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725560-5c4gd"] Jul 08 18:01:30.652474 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:01:30.652448 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725560-5c4gd"] Jul 08 18:01:30.957137 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:01:30.957053 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c7ac194-f1de-4fbc-bdd9-85501139b868" path="/var/lib/kubelet/pods/4c7ac194-f1de-4fbc-bdd9-85501139b868/volumes" Jul 08 18:02:11.166647 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:02:11.166610 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:03:44.933586 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:03:44.933560 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/1.log" Jul 08 18:03:44.936531 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:03:44.936508 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/1.log" Jul 08 18:04:11.826895 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:04:11.826721 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:11.913984 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:04:11.913935 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:25.274423 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:04:25.274389 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:34.904404 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:04:34.904354 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:43.190029 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:04:43.189984 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:48.514930 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:04:48.514896 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:51.434329 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:04:51.434286 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:51.537850 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:04:51.537812 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:51.921912 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:04:51.921882 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:58.515847 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:04:58.515811 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:58.954222 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:04:58.954186 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:59.014129 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:04:59.014080 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:04:59.943710 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:04:59.943673 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:00.041439 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.041405 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:00.236496 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.236346 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g"] Jul 08 18:05:00.241536 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.236786 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c7ac194-f1de-4fbc-bdd9-85501139b868" containerName="cleanup" Jul 08 18:05:00.241536 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.236800 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c7ac194-f1de-4fbc-bdd9-85501139b868" containerName="cleanup" Jul 08 18:05:00.241536 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.236881 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c7ac194-f1de-4fbc-bdd9-85501139b868" containerName="cleanup" Jul 08 18:05:00.241536 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.240672 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" Jul 08 18:05:00.242780 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.242758 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a114503a-serving-cert\"" Jul 08 18:05:00.252019 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.251992 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g"] Jul 08 18:05:00.283072 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.283032 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:00.331930 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.331895 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-maas-api-tls\") pod \"maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g\" (UID: \"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" Jul 08 18:05:00.332120 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.331937 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-openshift-service-ca\") pod \"maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g\" (UID: \"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" Jul 08 18:05:00.332120 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.332057 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bpzd\" (UniqueName: \"kubernetes.io/projected/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-kube-api-access-9bpzd\") pod \"maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g\" (UID: \"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" Jul 08 18:05:00.432543 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.432504 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-maas-api-tls\") pod \"maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g\" (UID: \"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" Jul 08 18:05:00.432543 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.432550 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-openshift-service-ca\") pod \"maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g\" (UID: \"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" Jul 08 18:05:00.432804 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.432673 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpzd\" (UniqueName: \"kubernetes.io/projected/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-kube-api-access-9bpzd\") pod \"maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g\" (UID: \"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" Jul 08 18:05:00.432804 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:05:00.432677 2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-serving-cert: secret "maas-api-e2e-ait-a114503a-serving-cert" not found Jul 08 18:05:00.432905 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:05:00.432806 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-maas-api-tls podName:e2e9ca6e-c738-43a7-aa68-9d866b0d0b12 nodeName:}" failed. No retries permitted until 2026-07-08 18:05:00.932783875 +0000 UTC m=+1576.616881133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-maas-api-tls") pod "maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" (UID: "e2e9ca6e-c738-43a7-aa68-9d866b0d0b12") : secret "maas-api-e2e-ait-a114503a-serving-cert" not found Jul 08 18:05:00.433203 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.433184 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-openshift-service-ca\") pod \"maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g\" (UID: \"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" Jul 08 18:05:00.444813 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.444780 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bpzd\" (UniqueName: \"kubernetes.io/projected/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-kube-api-access-9bpzd\") pod \"maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g\" (UID: \"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" Jul 08 18:05:00.643190 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.643157 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:00.857661 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.857623 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:00.938174 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.938093 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-maas-api-tls\") pod \"maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g\" (UID: \"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" Jul 08 18:05:00.940488 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.940458 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-maas-api-tls\") pod \"maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g\" (UID: \"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" Jul 08 18:05:00.960405 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:00.960373 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:01.152486 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:01.152453 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" Jul 08 18:05:01.297166 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:01.297137 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g"] Jul 08 18:05:01.298475 ip-10-0-129-226 kubenswrapper[2584]: W0708 18:05:01.298447 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e9ca6e_c738_43a7_aa68_9d866b0d0b12.slice/crio-a17fe238b49e77c4b908a51c2d255a8c97e4c7ff479d765ee8626c031ed35646 WatchSource:0}: Error finding container a17fe238b49e77c4b908a51c2d255a8c97e4c7ff479d765ee8626c031ed35646: Status 404 returned error can't find the container with id a17fe238b49e77c4b908a51c2d255a8c97e4c7ff479d765ee8626c031ed35646 Jul 08 18:05:01.299844 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:01.299826 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 18:05:01.524647 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:01.524567 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" event={"ID":"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12","Type":"ContainerStarted","Data":"a17fe238b49e77c4b908a51c2d255a8c97e4c7ff479d765ee8626c031ed35646"} Jul 08 18:05:02.533070 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:02.532965 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" event={"ID":"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12","Type":"ContainerStarted","Data":"2f2a9d29aa6e154fd9222ca93ce1366f71e8ab51a47ad01e12ec834cf324ae1a"} Jul 08 18:05:02.533070 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:02.533048 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" Jul 08 18:05:02.551712 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:02.551655 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" podStartSLOduration=1.590691584 podStartE2EDuration="2.551636476s" totalImagesPullingTime="960.944892ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:05:02.550513877 +0000 UTC m=+1578.234611165" watchObservedRunningTime="2026-07-08 18:05:02.551636476 +0000 UTC m=+1578.235733754" Jul 08 18:05:03.475967 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:03.475930 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:03.546697 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:03.546657 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:03.680268 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:03.680219 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:05.108326 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:05.108289 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:05.317263 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:05.317196 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:05.432597 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:05.432549 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:05.554474 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:05.554439 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g"] Jul 08 18:05:05.554704 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:05.554678 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" podUID="e2e9ca6e-c738-43a7-aa68-9d866b0d0b12" containerName="maas-api" containerID="cri-o://2f2a9d29aa6e154fd9222ca93ce1366f71e8ab51a47ad01e12ec834cf324ae1a" gracePeriod=30 Jul 08 18:05:05.819490 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:05.819461 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" Jul 08 18:05:05.897725 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:05.897687 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:05.987507 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:05.987417 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-maas-api-tls\") pod \"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12\" (UID: \"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12\") " Jul 08 18:05:05.987507 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:05.987501 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-openshift-service-ca\") pod \"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12\" (UID: \"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12\") " Jul 08 18:05:05.987750 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:05.987523 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bpzd\" (UniqueName: \"kubernetes.io/projected/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-kube-api-access-9bpzd\") pod \"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12\" (UID: \"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12\") " Jul 08 18:05:05.987873 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:05.987847 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e2e9ca6e-c738-43a7-aa68-9d866b0d0b12" (UID: "e2e9ca6e-c738-43a7-aa68-9d866b0d0b12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:05:05.989565 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:05.989541 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e2e9ca6e-c738-43a7-aa68-9d866b0d0b12" (UID: "e2e9ca6e-c738-43a7-aa68-9d866b0d0b12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:05:05.989651 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:05.989599 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-kube-api-access-9bpzd" (OuterVolumeSpecName: "kube-api-access-9bpzd") pod "e2e9ca6e-c738-43a7-aa68-9d866b0d0b12" (UID: "e2e9ca6e-c738-43a7-aa68-9d866b0d0b12"). InnerVolumeSpecName "kube-api-access-9bpzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:05:06.088306 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:06.088269 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-maas-api-tls\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:05:06.088306 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:06.088299 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-openshift-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:05:06.088306 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:06.088309 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bpzd\" (UniqueName: \"kubernetes.io/projected/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12-kube-api-access-9bpzd\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:05:06.556996 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:06.556957 2584 generic.go:356] "Generic (PLEG): container finished" podID="e2e9ca6e-c738-43a7-aa68-9d866b0d0b12" containerID="2f2a9d29aa6e154fd9222ca93ce1366f71e8ab51a47ad01e12ec834cf324ae1a" exitCode=0 Jul 08 18:05:06.556996 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:06.556999 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" Jul 08 18:05:06.557463 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:06.557043 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" event={"ID":"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12","Type":"ContainerDied","Data":"2f2a9d29aa6e154fd9222ca93ce1366f71e8ab51a47ad01e12ec834cf324ae1a"} Jul 08 18:05:06.557463 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:06.557098 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g" event={"ID":"e2e9ca6e-c738-43a7-aa68-9d866b0d0b12","Type":"ContainerDied","Data":"a17fe238b49e77c4b908a51c2d255a8c97e4c7ff479d765ee8626c031ed35646"} Jul 08 18:05:06.557463 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:06.557120 2584 scope.go:117] "RemoveContainer" containerID="2f2a9d29aa6e154fd9222ca93ce1366f71e8ab51a47ad01e12ec834cf324ae1a" Jul 08 18:05:06.567123 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:06.567104 2584 scope.go:117] "RemoveContainer" containerID="2f2a9d29aa6e154fd9222ca93ce1366f71e8ab51a47ad01e12ec834cf324ae1a" Jul 08 18:05:06.567415 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:05:06.567394 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f2a9d29aa6e154fd9222ca93ce1366f71e8ab51a47ad01e12ec834cf324ae1a\": container with ID starting with 2f2a9d29aa6e154fd9222ca93ce1366f71e8ab51a47ad01e12ec834cf324ae1a not found: ID does not exist" containerID="2f2a9d29aa6e154fd9222ca93ce1366f71e8ab51a47ad01e12ec834cf324ae1a" Jul 08 18:05:06.567499 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:06.567424 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f2a9d29aa6e154fd9222ca93ce1366f71e8ab51a47ad01e12ec834cf324ae1a"} err="failed to get container status \"2f2a9d29aa6e154fd9222ca93ce1366f71e8ab51a47ad01e12ec834cf324ae1a\": rpc error: code = NotFound desc = could not find container \"2f2a9d29aa6e154fd9222ca93ce1366f71e8ab51a47ad01e12ec834cf324ae1a\": container with ID starting with 2f2a9d29aa6e154fd9222ca93ce1366f71e8ab51a47ad01e12ec834cf324ae1a not found: ID does not exist" Jul 08 18:05:06.580549 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:06.580513 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g"] Jul 08 18:05:06.585972 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:06.585938 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a114503a-d9b4458c4-4cd6g"] Jul 08 18:05:06.964009 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:06.963970 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2e9ca6e-c738-43a7-aa68-9d866b0d0b12" path="/var/lib/kubelet/pods/e2e9ca6e-c738-43a7-aa68-9d866b0d0b12/volumes" Jul 08 18:05:07.748976 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:07.748944 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:11.941089 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:11.941051 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:12.000645 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:12.000607 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:12.168974 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:12.168938 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:12.280969 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:12.280872 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:13.290876 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.290831 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:13.387886 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.387843 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:13.484194 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.484151 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q"] Jul 08 18:05:13.484772 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.484748 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2e9ca6e-c738-43a7-aa68-9d866b0d0b12" containerName="maas-api" Jul 08 18:05:13.484772 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.484774 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2e9ca6e-c738-43a7-aa68-9d866b0d0b12" containerName="maas-api" Jul 08 18:05:13.484933 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.484894 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2e9ca6e-c738-43a7-aa68-9d866b0d0b12" containerName="maas-api" Jul 08 18:05:13.489690 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.489663 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" Jul 08 18:05:13.492183 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.492160 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9c953ff5-serving-cert\"" Jul 08 18:05:13.500229 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.500204 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q"] Jul 08 18:05:13.537284 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.537230 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:13.550493 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.550410 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f279m\" (UniqueName: \"kubernetes.io/projected/5d1bfd1b-99df-450e-ae62-4263d7a0ba50-kube-api-access-f279m\") pod \"maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q\" (UID: \"5d1bfd1b-99df-450e-ae62-4263d7a0ba50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" Jul 08 18:05:13.550493 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.550459 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d1bfd1b-99df-450e-ae62-4263d7a0ba50-maas-api-tls\") pod \"maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q\" (UID: \"5d1bfd1b-99df-450e-ae62-4263d7a0ba50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" Jul 08 18:05:13.550678 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.550583 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1bfd1b-99df-450e-ae62-4263d7a0ba50-openshift-service-ca\") pod \"maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q\" (UID: \"5d1bfd1b-99df-450e-ae62-4263d7a0ba50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" Jul 08 18:05:13.651952 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.651910 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1bfd1b-99df-450e-ae62-4263d7a0ba50-openshift-service-ca\") pod \"maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q\" (UID: \"5d1bfd1b-99df-450e-ae62-4263d7a0ba50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" Jul 08 18:05:13.652158 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.651964 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f279m\" (UniqueName: \"kubernetes.io/projected/5d1bfd1b-99df-450e-ae62-4263d7a0ba50-kube-api-access-f279m\") pod \"maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q\" (UID: \"5d1bfd1b-99df-450e-ae62-4263d7a0ba50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" Jul 08 18:05:13.652158 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.652007 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d1bfd1b-99df-450e-ae62-4263d7a0ba50-maas-api-tls\") pod \"maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q\" (UID: \"5d1bfd1b-99df-450e-ae62-4263d7a0ba50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" Jul 08 18:05:13.652709 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.652681 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1bfd1b-99df-450e-ae62-4263d7a0ba50-openshift-service-ca\") pod \"maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q\" (UID: \"5d1bfd1b-99df-450e-ae62-4263d7a0ba50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" Jul 08 18:05:13.654506 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.654485 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d1bfd1b-99df-450e-ae62-4263d7a0ba50-maas-api-tls\") pod \"maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q\" (UID: \"5d1bfd1b-99df-450e-ae62-4263d7a0ba50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" Jul 08 18:05:13.665280 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.665229 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f279m\" (UniqueName: \"kubernetes.io/projected/5d1bfd1b-99df-450e-ae62-4263d7a0ba50-kube-api-access-f279m\") pod \"maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q\" (UID: \"5d1bfd1b-99df-450e-ae62-4263d7a0ba50\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" Jul 08 18:05:13.682155 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.682113 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:13.802882 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.802779 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" Jul 08 18:05:13.951688 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.951659 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q"] Jul 08 18:05:13.953037 ip-10-0-129-226 kubenswrapper[2584]: W0708 18:05:13.953012 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1bfd1b_99df_450e_ae62_4263d7a0ba50.slice/crio-b91ad12472b7e83a34ba784c853cc10c954a649ac7045c2145bdd971dfa4e367 WatchSource:0}: Error finding container b91ad12472b7e83a34ba784c853cc10c954a649ac7045c2145bdd971dfa4e367: Status 404 returned error can't find the container with id b91ad12472b7e83a34ba784c853cc10c954a649ac7045c2145bdd971dfa4e367 Jul 08 18:05:13.991063 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:13.991023 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:14.094720 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:14.094622 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:14.604644 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:14.604611 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" event={"ID":"5d1bfd1b-99df-450e-ae62-4263d7a0ba50","Type":"ContainerStarted","Data":"b825eba806d3f53d4970f98a26be5d1767ebbc55733979772f643eec6fa01d62"} Jul 08 18:05:14.605091 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:14.604663 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" event={"ID":"5d1bfd1b-99df-450e-ae62-4263d7a0ba50","Type":"ContainerStarted","Data":"b91ad12472b7e83a34ba784c853cc10c954a649ac7045c2145bdd971dfa4e367"} Jul 08 18:05:14.605091 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:14.604684 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" Jul 08 18:05:14.624382 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:14.624331 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" podStartSLOduration=1.1379333680000001 podStartE2EDuration="1.624312588s" totalImagesPullingTime="486.37922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:05:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:05:14.623930749 +0000 UTC m=+1590.308028026" watchObservedRunningTime="2026-07-08 18:05:14.624312588 +0000 UTC m=+1590.308409866" Jul 08 18:05:15.495459 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:15.495422 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:18.535724 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:18.535687 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:18.974669 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:18.974628 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q"] Jul 08 18:05:18.974887 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:18.974852 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" podUID="5d1bfd1b-99df-450e-ae62-4263d7a0ba50" containerName="maas-api" containerID="cri-o://b825eba806d3f53d4970f98a26be5d1767ebbc55733979772f643eec6fa01d62" gracePeriod=30 Jul 08 18:05:18.997181 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:18.997145 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:19.226288 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:19.226210 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" Jul 08 18:05:19.306973 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:19.306939 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f279m\" (UniqueName: \"kubernetes.io/projected/5d1bfd1b-99df-450e-ae62-4263d7a0ba50-kube-api-access-f279m\") pod \"5d1bfd1b-99df-450e-ae62-4263d7a0ba50\" (UID: \"5d1bfd1b-99df-450e-ae62-4263d7a0ba50\") " Jul 08 18:05:19.306973 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:19.306974 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d1bfd1b-99df-450e-ae62-4263d7a0ba50-maas-api-tls\") pod \"5d1bfd1b-99df-450e-ae62-4263d7a0ba50\" (UID: \"5d1bfd1b-99df-450e-ae62-4263d7a0ba50\") " Jul 08 18:05:19.307268 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:19.307017 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1bfd1b-99df-450e-ae62-4263d7a0ba50-openshift-service-ca\") pod \"5d1bfd1b-99df-450e-ae62-4263d7a0ba50\" (UID: \"5d1bfd1b-99df-450e-ae62-4263d7a0ba50\") " Jul 08 18:05:19.307477 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:19.307451 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1bfd1b-99df-450e-ae62-4263d7a0ba50-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5d1bfd1b-99df-450e-ae62-4263d7a0ba50" (UID: "5d1bfd1b-99df-450e-ae62-4263d7a0ba50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:05:19.308981 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:19.308955 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1bfd1b-99df-450e-ae62-4263d7a0ba50-kube-api-access-f279m" (OuterVolumeSpecName: "kube-api-access-f279m") pod "5d1bfd1b-99df-450e-ae62-4263d7a0ba50" (UID: "5d1bfd1b-99df-450e-ae62-4263d7a0ba50"). InnerVolumeSpecName "kube-api-access-f279m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:05:19.309059 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:19.309040 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d1bfd1b-99df-450e-ae62-4263d7a0ba50-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5d1bfd1b-99df-450e-ae62-4263d7a0ba50" (UID: "5d1bfd1b-99df-450e-ae62-4263d7a0ba50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:05:19.408396 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:19.408353 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d1bfd1b-99df-450e-ae62-4263d7a0ba50-openshift-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:05:19.408396 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:19.408389 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f279m\" (UniqueName: \"kubernetes.io/projected/5d1bfd1b-99df-450e-ae62-4263d7a0ba50-kube-api-access-f279m\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:05:19.408396 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:19.408399 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d1bfd1b-99df-450e-ae62-4263d7a0ba50-maas-api-tls\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:05:19.635540 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:19.635501 2584 generic.go:356] "Generic (PLEG): container finished" podID="5d1bfd1b-99df-450e-ae62-4263d7a0ba50" containerID="b825eba806d3f53d4970f98a26be5d1767ebbc55733979772f643eec6fa01d62" exitCode=0 Jul 08 18:05:19.636061 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:19.635551 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" Jul 08 18:05:19.636061 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:19.635577 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" event={"ID":"5d1bfd1b-99df-450e-ae62-4263d7a0ba50","Type":"ContainerDied","Data":"b825eba806d3f53d4970f98a26be5d1767ebbc55733979772f643eec6fa01d62"} Jul 08 18:05:19.636061 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:19.635638 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q" event={"ID":"5d1bfd1b-99df-450e-ae62-4263d7a0ba50","Type":"ContainerDied","Data":"b91ad12472b7e83a34ba784c853cc10c954a649ac7045c2145bdd971dfa4e367"} Jul 08 18:05:19.636061 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:19.635668 2584 scope.go:117] "RemoveContainer" containerID="b825eba806d3f53d4970f98a26be5d1767ebbc55733979772f643eec6fa01d62" Jul 08 18:05:19.649911 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:19.649889 2584 scope.go:117] "RemoveContainer" containerID="b825eba806d3f53d4970f98a26be5d1767ebbc55733979772f643eec6fa01d62" Jul 08 18:05:19.650419 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:05:19.650388 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b825eba806d3f53d4970f98a26be5d1767ebbc55733979772f643eec6fa01d62\": container with ID starting with b825eba806d3f53d4970f98a26be5d1767ebbc55733979772f643eec6fa01d62 not found: ID does not exist" containerID="b825eba806d3f53d4970f98a26be5d1767ebbc55733979772f643eec6fa01d62" Jul 08 18:05:19.650535 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:19.650429 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b825eba806d3f53d4970f98a26be5d1767ebbc55733979772f643eec6fa01d62"} err="failed to get container status \"b825eba806d3f53d4970f98a26be5d1767ebbc55733979772f643eec6fa01d62\": rpc error: code = NotFound desc = could not find container \"b825eba806d3f53d4970f98a26be5d1767ebbc55733979772f643eec6fa01d62\": container with ID starting with b825eba806d3f53d4970f98a26be5d1767ebbc55733979772f643eec6fa01d62 not found: ID does not exist" Jul 08 18:05:19.662145 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:19.662114 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q"] Jul 08 18:05:19.667073 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:19.667047 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9c953ff5-84df9fc898-bwc7q"] Jul 08 18:05:20.956503 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:20.956469 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d1bfd1b-99df-450e-ae62-4263d7a0ba50" path="/var/lib/kubelet/pods/5d1bfd1b-99df-450e-ae62-4263d7a0ba50/volumes" Jul 08 18:05:24.714231 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:24.714195 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:24.817625 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:24.817586 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:27.079575 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:27.079535 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:31.536275 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:31.536213 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:31.638284 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:31.638223 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:31.765788 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:31.765743 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:31.830119 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:31.829984 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:32.888860 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:32.888811 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:32.967042 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:32.966994 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:33.101055 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.101011 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk"] Jul 08 18:05:33.101672 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.101645 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d1bfd1b-99df-450e-ae62-4263d7a0ba50" containerName="maas-api" Jul 08 18:05:33.101672 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.101671 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d1bfd1b-99df-450e-ae62-4263d7a0ba50" containerName="maas-api" Jul 08 18:05:33.101823 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.101785 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d1bfd1b-99df-450e-ae62-4263d7a0ba50" containerName="maas-api" Jul 08 18:05:33.106700 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.106669 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" Jul 08 18:05:33.109145 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.109125 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-95e2b8a6-serving-cert\"" Jul 08 18:05:33.116146 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.116122 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk"] Jul 08 18:05:33.133077 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.133045 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:33.236918 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.236821 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wstnr\" (UniqueName: \"kubernetes.io/projected/e0c4e059-b891-4ca8-832a-d32bd7d44e70-kube-api-access-wstnr\") pod \"maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk\" (UID: \"e0c4e059-b891-4ca8-832a-d32bd7d44e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" Jul 08 18:05:33.236918 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.236861 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c4e059-b891-4ca8-832a-d32bd7d44e70-maas-api-tls\") pod \"maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk\" (UID: \"e0c4e059-b891-4ca8-832a-d32bd7d44e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" Jul 08 18:05:33.236918 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.236880 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c4e059-b891-4ca8-832a-d32bd7d44e70-openshift-service-ca\") pod \"maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk\" (UID: \"e0c4e059-b891-4ca8-832a-d32bd7d44e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" Jul 08 18:05:33.338199 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.338159 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wstnr\" (UniqueName: \"kubernetes.io/projected/e0c4e059-b891-4ca8-832a-d32bd7d44e70-kube-api-access-wstnr\") pod \"maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk\" (UID: \"e0c4e059-b891-4ca8-832a-d32bd7d44e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" Jul 08 18:05:33.338363 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.338210 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c4e059-b891-4ca8-832a-d32bd7d44e70-maas-api-tls\") pod \"maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk\" (UID: \"e0c4e059-b891-4ca8-832a-d32bd7d44e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" Jul 08 18:05:33.338363 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.338254 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c4e059-b891-4ca8-832a-d32bd7d44e70-openshift-service-ca\") pod \"maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk\" (UID: \"e0c4e059-b891-4ca8-832a-d32bd7d44e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" Jul 08 18:05:33.339096 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.339071 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c4e059-b891-4ca8-832a-d32bd7d44e70-openshift-service-ca\") pod \"maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk\" (UID: \"e0c4e059-b891-4ca8-832a-d32bd7d44e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" Jul 08 18:05:33.340858 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.340839 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c4e059-b891-4ca8-832a-d32bd7d44e70-maas-api-tls\") pod \"maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk\" (UID: \"e0c4e059-b891-4ca8-832a-d32bd7d44e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" Jul 08 18:05:33.349000 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.348977 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wstnr\" (UniqueName: \"kubernetes.io/projected/e0c4e059-b891-4ca8-832a-d32bd7d44e70-kube-api-access-wstnr\") pod \"maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk\" (UID: \"e0c4e059-b891-4ca8-832a-d32bd7d44e70\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" Jul 08 18:05:33.413547 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.413511 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:33.418334 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.418310 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" Jul 08 18:05:33.556924 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.556891 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk"] Jul 08 18:05:33.558812 ip-10-0-129-226 kubenswrapper[2584]: W0708 18:05:33.558785 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c4e059_b891_4ca8_832a_d32bd7d44e70.slice/crio-806b7f80ed7955158fb9899d4a401fc8c450204ea0dd787ea6e1bec96a1f1b08 WatchSource:0}: Error finding container 806b7f80ed7955158fb9899d4a401fc8c450204ea0dd787ea6e1bec96a1f1b08: Status 404 returned error can't find the container with id 806b7f80ed7955158fb9899d4a401fc8c450204ea0dd787ea6e1bec96a1f1b08 Jul 08 18:05:33.715757 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.715716 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" event={"ID":"e0c4e059-b891-4ca8-832a-d32bd7d44e70","Type":"ContainerStarted","Data":"806b7f80ed7955158fb9899d4a401fc8c450204ea0dd787ea6e1bec96a1f1b08"} Jul 08 18:05:33.779768 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.779669 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:33.876495 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:33.876450 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:34.722418 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:34.722384 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" event={"ID":"e0c4e059-b891-4ca8-832a-d32bd7d44e70","Type":"ContainerStarted","Data":"684743d292e3e38a7a38ae0297f5f651ef9491a49bb6e72aa893abf0f8d7b47c"} Jul 08 18:05:34.722789 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:34.722469 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" Jul 08 18:05:34.739999 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:34.739942 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" podStartSLOduration=1.411737525 podStartE2EDuration="1.739921542s" totalImagesPullingTime="328.184017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:05:34.739047312 +0000 UTC m=+1610.423144584" watchObservedRunningTime="2026-07-08 18:05:34.739921542 +0000 UTC m=+1610.424018820" Jul 08 18:05:35.598315 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:35.598267 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:37.378178 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:37.378087 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:37.621755 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:37.621716 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:37.734351 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:37.734231 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:37.892118 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:37.892085 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk"] Jul 08 18:05:37.892385 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:37.892346 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" podUID="e0c4e059-b891-4ca8-832a-d32bd7d44e70" containerName="maas-api" containerID="cri-o://684743d292e3e38a7a38ae0297f5f651ef9491a49bb6e72aa893abf0f8d7b47c" gracePeriod=30 Jul 08 18:05:37.922037 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:37.921616 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:38.155885 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.155863 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" Jul 08 18:05:38.282828 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.282745 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c4e059-b891-4ca8-832a-d32bd7d44e70-openshift-service-ca\") pod \"e0c4e059-b891-4ca8-832a-d32bd7d44e70\" (UID: \"e0c4e059-b891-4ca8-832a-d32bd7d44e70\") " Jul 08 18:05:38.282987 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.282828 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wstnr\" (UniqueName: \"kubernetes.io/projected/e0c4e059-b891-4ca8-832a-d32bd7d44e70-kube-api-access-wstnr\") pod \"e0c4e059-b891-4ca8-832a-d32bd7d44e70\" (UID: \"e0c4e059-b891-4ca8-832a-d32bd7d44e70\") " Jul 08 18:05:38.282987 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.282879 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c4e059-b891-4ca8-832a-d32bd7d44e70-maas-api-tls\") pod \"e0c4e059-b891-4ca8-832a-d32bd7d44e70\" (UID: \"e0c4e059-b891-4ca8-832a-d32bd7d44e70\") " Jul 08 18:05:38.283120 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.283083 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c4e059-b891-4ca8-832a-d32bd7d44e70-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e0c4e059-b891-4ca8-832a-d32bd7d44e70" (UID: "e0c4e059-b891-4ca8-832a-d32bd7d44e70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:05:38.284916 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.284893 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c4e059-b891-4ca8-832a-d32bd7d44e70-kube-api-access-wstnr" (OuterVolumeSpecName: "kube-api-access-wstnr") pod "e0c4e059-b891-4ca8-832a-d32bd7d44e70" (UID: "e0c4e059-b891-4ca8-832a-d32bd7d44e70"). InnerVolumeSpecName "kube-api-access-wstnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:05:38.285002 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.284917 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0c4e059-b891-4ca8-832a-d32bd7d44e70-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e0c4e059-b891-4ca8-832a-d32bd7d44e70" (UID: "e0c4e059-b891-4ca8-832a-d32bd7d44e70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:05:38.383640 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.383605 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0c4e059-b891-4ca8-832a-d32bd7d44e70-maas-api-tls\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:05:38.383640 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.383634 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c4e059-b891-4ca8-832a-d32bd7d44e70-openshift-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:05:38.383640 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.383643 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wstnr\" (UniqueName: \"kubernetes.io/projected/e0c4e059-b891-4ca8-832a-d32bd7d44e70-kube-api-access-wstnr\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:05:38.745736 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.745699 2584 generic.go:356] "Generic (PLEG): container finished" podID="e0c4e059-b891-4ca8-832a-d32bd7d44e70" containerID="684743d292e3e38a7a38ae0297f5f651ef9491a49bb6e72aa893abf0f8d7b47c" exitCode=0 Jul 08 18:05:38.745922 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.745758 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" Jul 08 18:05:38.745922 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.745790 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" event={"ID":"e0c4e059-b891-4ca8-832a-d32bd7d44e70","Type":"ContainerDied","Data":"684743d292e3e38a7a38ae0297f5f651ef9491a49bb6e72aa893abf0f8d7b47c"} Jul 08 18:05:38.745922 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.745881 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk" event={"ID":"e0c4e059-b891-4ca8-832a-d32bd7d44e70","Type":"ContainerDied","Data":"806b7f80ed7955158fb9899d4a401fc8c450204ea0dd787ea6e1bec96a1f1b08"} Jul 08 18:05:38.745922 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.745904 2584 scope.go:117] "RemoveContainer" containerID="684743d292e3e38a7a38ae0297f5f651ef9491a49bb6e72aa893abf0f8d7b47c" Jul 08 18:05:38.755597 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.755577 2584 scope.go:117] "RemoveContainer" containerID="684743d292e3e38a7a38ae0297f5f651ef9491a49bb6e72aa893abf0f8d7b47c" Jul 08 18:05:38.755835 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:05:38.755817 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684743d292e3e38a7a38ae0297f5f651ef9491a49bb6e72aa893abf0f8d7b47c\": container with ID starting with 684743d292e3e38a7a38ae0297f5f651ef9491a49bb6e72aa893abf0f8d7b47c not found: ID does not exist" containerID="684743d292e3e38a7a38ae0297f5f651ef9491a49bb6e72aa893abf0f8d7b47c" Jul 08 18:05:38.755887 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.755842 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684743d292e3e38a7a38ae0297f5f651ef9491a49bb6e72aa893abf0f8d7b47c"} err="failed to get container status \"684743d292e3e38a7a38ae0297f5f651ef9491a49bb6e72aa893abf0f8d7b47c\": rpc error: code = NotFound desc = could not find container \"684743d292e3e38a7a38ae0297f5f651ef9491a49bb6e72aa893abf0f8d7b47c\": container with ID starting with 684743d292e3e38a7a38ae0297f5f651ef9491a49bb6e72aa893abf0f8d7b47c not found: ID does not exist" Jul 08 18:05:38.769983 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.769960 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk"] Jul 08 18:05:38.775181 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.775161 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-95e2b8a6-67ddf59b67-7wwtk"] Jul 08 18:05:38.957004 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:38.956970 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0c4e059-b891-4ca8-832a-d32bd7d44e70" path="/var/lib/kubelet/pods/e0c4e059-b891-4ca8-832a-d32bd7d44e70/volumes" Jul 08 18:05:39.607338 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:39.607303 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:43.977226 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:43.977158 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:44.034209 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:44.034171 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:44.470049 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:44.470009 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:44.570181 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:44.570139 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:44.673169 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:44.673129 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:46.197054 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:46.197017 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:52.828952 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:52.828906 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:52.932210 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:52.932166 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:53.029231 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:53.029192 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:53.130715 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:53.130681 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:05:53.655859 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:05:53.655821 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:19.685741 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:06:19.685705 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:25.984550 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:06:25.984513 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:28.150264 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:06:28.150204 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:42.157206 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:06:42.157155 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:48.839605 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:06:48.839567 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:48.945426 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:06:48.945391 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:49.044476 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:06:49.044440 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:06:59.633337 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:06:59.633298 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:00.466447 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:00.466406 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:07.388000 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:07.387892 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:07.491756 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:07.491717 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:07.589787 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:07.589757 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:07.688355 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:07.688226 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:08.539759 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:08.539717 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:08.634285 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:08.634232 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:08.736468 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:08.736430 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:10.458416 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:10.458379 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:17.026237 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:17.026198 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:17.127210 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:17.127177 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:17.733798 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:17.733764 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:24.799660 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:24.799621 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:24.898081 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:24.898046 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:25.060898 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:25.060802 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:25.380867 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:25.380830 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:25.514476 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:25.514436 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:28.506919 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:28.506883 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:28.646519 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:28.646486 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t"] Jul 08 18:07:28.646939 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:28.646920 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0c4e059-b891-4ca8-832a-d32bd7d44e70" containerName="maas-api" Jul 08 18:07:28.647043 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:28.646941 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0c4e059-b891-4ca8-832a-d32bd7d44e70" containerName="maas-api" Jul 08 18:07:28.647104 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:28.647055 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0c4e059-b891-4ca8-832a-d32bd7d44e70" containerName="maas-api" Jul 08 18:07:28.650298 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:28.650277 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" Jul 08 18:07:28.652906 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:28.652882 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e894b595-serving-cert\"" Jul 08 18:07:28.663062 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:28.663039 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t"] Jul 08 18:07:28.734286 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:28.734222 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fxx2\" (UniqueName: \"kubernetes.io/projected/66400d6c-7ec8-4170-8a8e-fa42352f3919-kube-api-access-7fxx2\") pod \"maas-api-e2e-mt-e894b595-5697c68976-9446t\" (UID: \"66400d6c-7ec8-4170-8a8e-fa42352f3919\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" Jul 08 18:07:28.734479 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:28.734321 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66400d6c-7ec8-4170-8a8e-fa42352f3919-maas-api-tls\") pod \"maas-api-e2e-mt-e894b595-5697c68976-9446t\" (UID: \"66400d6c-7ec8-4170-8a8e-fa42352f3919\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" Jul 08 18:07:28.734479 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:28.734369 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66400d6c-7ec8-4170-8a8e-fa42352f3919-openshift-service-ca\") pod \"maas-api-e2e-mt-e894b595-5697c68976-9446t\" (UID: \"66400d6c-7ec8-4170-8a8e-fa42352f3919\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" Jul 08 18:07:28.835759 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:28.835657 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fxx2\" (UniqueName: \"kubernetes.io/projected/66400d6c-7ec8-4170-8a8e-fa42352f3919-kube-api-access-7fxx2\") pod \"maas-api-e2e-mt-e894b595-5697c68976-9446t\" (UID: \"66400d6c-7ec8-4170-8a8e-fa42352f3919\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" Jul 08 18:07:28.835759 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:28.835715 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66400d6c-7ec8-4170-8a8e-fa42352f3919-maas-api-tls\") pod \"maas-api-e2e-mt-e894b595-5697c68976-9446t\" (UID: \"66400d6c-7ec8-4170-8a8e-fa42352f3919\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" Jul 08 18:07:28.835759 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:28.835757 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66400d6c-7ec8-4170-8a8e-fa42352f3919-openshift-service-ca\") pod \"maas-api-e2e-mt-e894b595-5697c68976-9446t\" (UID: \"66400d6c-7ec8-4170-8a8e-fa42352f3919\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" Jul 08 18:07:28.836075 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:07:28.835879 2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-serving-cert: secret "maas-api-e2e-mt-e894b595-serving-cert" not found Jul 08 18:07:28.836075 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:07:28.835955 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66400d6c-7ec8-4170-8a8e-fa42352f3919-maas-api-tls podName:66400d6c-7ec8-4170-8a8e-fa42352f3919 nodeName:}" failed. No retries permitted until 2026-07-08 18:07:29.335932343 +0000 UTC m=+1725.020029607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/66400d6c-7ec8-4170-8a8e-fa42352f3919-maas-api-tls") pod "maas-api-e2e-mt-e894b595-5697c68976-9446t" (UID: "66400d6c-7ec8-4170-8a8e-fa42352f3919") : secret "maas-api-e2e-mt-e894b595-serving-cert" not found Jul 08 18:07:28.836552 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:28.836532 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66400d6c-7ec8-4170-8a8e-fa42352f3919-openshift-service-ca\") pod \"maas-api-e2e-mt-e894b595-5697c68976-9446t\" (UID: \"66400d6c-7ec8-4170-8a8e-fa42352f3919\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" Jul 08 18:07:28.846438 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:28.846415 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fxx2\" (UniqueName: \"kubernetes.io/projected/66400d6c-7ec8-4170-8a8e-fa42352f3919-kube-api-access-7fxx2\") pod \"maas-api-e2e-mt-e894b595-5697c68976-9446t\" (UID: \"66400d6c-7ec8-4170-8a8e-fa42352f3919\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" Jul 08 18:07:29.339681 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:29.339645 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66400d6c-7ec8-4170-8a8e-fa42352f3919-maas-api-tls\") pod \"maas-api-e2e-mt-e894b595-5697c68976-9446t\" (UID: \"66400d6c-7ec8-4170-8a8e-fa42352f3919\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" Jul 08 18:07:29.342165 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:29.342136 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66400d6c-7ec8-4170-8a8e-fa42352f3919-maas-api-tls\") pod \"maas-api-e2e-mt-e894b595-5697c68976-9446t\" (UID: \"66400d6c-7ec8-4170-8a8e-fa42352f3919\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" Jul 08 18:07:29.560409 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:29.560366 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" Jul 08 18:07:29.725413 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:29.725387 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t"] Jul 08 18:07:29.727483 ip-10-0-129-226 kubenswrapper[2584]: W0708 18:07:29.727440 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66400d6c_7ec8_4170_8a8e_fa42352f3919.slice/crio-9e4f1954c7e37d81ef6d3e673f9be85fb76cc88dbf6688bc668e9a37890b885b WatchSource:0}: Error finding container 9e4f1954c7e37d81ef6d3e673f9be85fb76cc88dbf6688bc668e9a37890b885b: Status 404 returned error can't find the container with id 9e4f1954c7e37d81ef6d3e673f9be85fb76cc88dbf6688bc668e9a37890b885b Jul 08 18:07:29.864640 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:29.864606 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:30.022195 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:30.022081 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:30.370463 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:30.370431 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" event={"ID":"66400d6c-7ec8-4170-8a8e-fa42352f3919","Type":"ContainerStarted","Data":"9e4f1954c7e37d81ef6d3e673f9be85fb76cc88dbf6688bc668e9a37890b885b"} Jul 08 18:07:32.383206 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:32.383169 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" event={"ID":"66400d6c-7ec8-4170-8a8e-fa42352f3919","Type":"ContainerStarted","Data":"9559411e6794e1ae885e3d56d41a3f266c2ec37b34bc3a27109019ca70361aaf"} Jul 08 18:07:32.383675 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:32.383312 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" Jul 08 18:07:32.418400 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:32.418351 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" podStartSLOduration=2.5634490039999998 podStartE2EDuration="4.418336934s" totalImagesPullingTime="1.85488793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:07:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:07:32.416225233 +0000 UTC m=+1728.100322511" watchObservedRunningTime="2026-07-08 18:07:32.418336934 +0000 UTC m=+1728.102434215" Jul 08 18:07:33.085657 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:33.085620 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:33.784409 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:33.784364 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:33.904209 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:33.904168 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:34.286466 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:34.286430 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:38.396218 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:38.396185 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" Jul 08 18:07:41.342935 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:41.342903 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:41.436822 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:41.436780 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:41.887699 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:41.887657 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:42.384455 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:42.384423 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t"] Jul 08 18:07:42.384921 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:42.384609 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" podUID="66400d6c-7ec8-4170-8a8e-fa42352f3919" containerName="maas-api" containerID="cri-o://9559411e6794e1ae885e3d56d41a3f266c2ec37b34bc3a27109019ca70361aaf" gracePeriod=30 Jul 08 18:07:42.632950 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:42.632923 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" Jul 08 18:07:42.762280 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:42.762160 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66400d6c-7ec8-4170-8a8e-fa42352f3919-openshift-service-ca\") pod \"66400d6c-7ec8-4170-8a8e-fa42352f3919\" (UID: \"66400d6c-7ec8-4170-8a8e-fa42352f3919\") " Jul 08 18:07:42.762280 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:42.762215 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fxx2\" (UniqueName: \"kubernetes.io/projected/66400d6c-7ec8-4170-8a8e-fa42352f3919-kube-api-access-7fxx2\") pod \"66400d6c-7ec8-4170-8a8e-fa42352f3919\" (UID: \"66400d6c-7ec8-4170-8a8e-fa42352f3919\") " Jul 08 18:07:42.762477 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:42.762347 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66400d6c-7ec8-4170-8a8e-fa42352f3919-maas-api-tls\") pod \"66400d6c-7ec8-4170-8a8e-fa42352f3919\" (UID: \"66400d6c-7ec8-4170-8a8e-fa42352f3919\") " Jul 08 18:07:42.762613 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:42.762578 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66400d6c-7ec8-4170-8a8e-fa42352f3919-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "66400d6c-7ec8-4170-8a8e-fa42352f3919" (UID: "66400d6c-7ec8-4170-8a8e-fa42352f3919"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:07:42.764333 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:42.764313 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66400d6c-7ec8-4170-8a8e-fa42352f3919-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "66400d6c-7ec8-4170-8a8e-fa42352f3919" (UID: "66400d6c-7ec8-4170-8a8e-fa42352f3919"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:07:42.764450 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:42.764429 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66400d6c-7ec8-4170-8a8e-fa42352f3919-kube-api-access-7fxx2" (OuterVolumeSpecName: "kube-api-access-7fxx2") pod "66400d6c-7ec8-4170-8a8e-fa42352f3919" (UID: "66400d6c-7ec8-4170-8a8e-fa42352f3919"). InnerVolumeSpecName "kube-api-access-7fxx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:07:42.863078 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:42.863046 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fxx2\" (UniqueName: \"kubernetes.io/projected/66400d6c-7ec8-4170-8a8e-fa42352f3919-kube-api-access-7fxx2\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:07:42.863078 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:42.863076 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66400d6c-7ec8-4170-8a8e-fa42352f3919-maas-api-tls\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:07:42.863365 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:42.863089 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66400d6c-7ec8-4170-8a8e-fa42352f3919-openshift-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:07:43.330974 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:43.330931 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:43.431969 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:43.431934 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:43.451987 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:43.451947 2584 generic.go:356] "Generic (PLEG): container finished" podID="66400d6c-7ec8-4170-8a8e-fa42352f3919" containerID="9559411e6794e1ae885e3d56d41a3f266c2ec37b34bc3a27109019ca70361aaf" exitCode=0 Jul 08 18:07:43.452161 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:43.452012 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" Jul 08 18:07:43.452161 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:43.452031 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" event={"ID":"66400d6c-7ec8-4170-8a8e-fa42352f3919","Type":"ContainerDied","Data":"9559411e6794e1ae885e3d56d41a3f266c2ec37b34bc3a27109019ca70361aaf"} Jul 08 18:07:43.452161 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:43.452115 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t" event={"ID":"66400d6c-7ec8-4170-8a8e-fa42352f3919","Type":"ContainerDied","Data":"9e4f1954c7e37d81ef6d3e673f9be85fb76cc88dbf6688bc668e9a37890b885b"} Jul 08 18:07:43.452161 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:43.452140 2584 scope.go:117] "RemoveContainer" containerID="9559411e6794e1ae885e3d56d41a3f266c2ec37b34bc3a27109019ca70361aaf" Jul 08 18:07:43.461127 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:43.461110 2584 scope.go:117] "RemoveContainer" containerID="9559411e6794e1ae885e3d56d41a3f266c2ec37b34bc3a27109019ca70361aaf" Jul 08 18:07:43.461404 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:07:43.461383 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9559411e6794e1ae885e3d56d41a3f266c2ec37b34bc3a27109019ca70361aaf\": container with ID starting with 9559411e6794e1ae885e3d56d41a3f266c2ec37b34bc3a27109019ca70361aaf not found: ID does not exist" containerID="9559411e6794e1ae885e3d56d41a3f266c2ec37b34bc3a27109019ca70361aaf" Jul 08 18:07:43.461492 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:43.461411 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9559411e6794e1ae885e3d56d41a3f266c2ec37b34bc3a27109019ca70361aaf"} err="failed to get container status \"9559411e6794e1ae885e3d56d41a3f266c2ec37b34bc3a27109019ca70361aaf\": rpc error: code = NotFound desc = could not find container \"9559411e6794e1ae885e3d56d41a3f266c2ec37b34bc3a27109019ca70361aaf\": container with ID starting with 9559411e6794e1ae885e3d56d41a3f266c2ec37b34bc3a27109019ca70361aaf not found: ID does not exist" Jul 08 18:07:43.480097 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:43.480065 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t"] Jul 08 18:07:43.487282 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:43.487256 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e894b595-5697c68976-9446t"] Jul 08 18:07:44.958195 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:44.958159 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66400d6c-7ec8-4170-8a8e-fa42352f3919" path="/var/lib/kubelet/pods/66400d6c-7ec8-4170-8a8e-fa42352f3919/volumes" Jul 08 18:07:50.004725 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:50.004692 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:50.112139 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:50.112099 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:50.317342 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:50.317232 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:51.474805 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:51.474764 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:51.551620 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:51.551586 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:56.834428 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:56.834386 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:56.932670 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:56.932629 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:58.858735 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:58.858699 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:58.984894 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:58.984856 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:59.070961 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:59.070916 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:07:59.242080 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:07:59.242040 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:06.266828 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:06.266788 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:06.332725 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:06.332689 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:13.397767 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:13.397728 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:14.171940 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:14.171901 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:14.382821 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:14.382779 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:17.956032 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:17.955982 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:18.048526 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.048484 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:18.076092 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.076060 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85"] Jul 08 18:08:18.076528 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.076513 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66400d6c-7ec8-4170-8a8e-fa42352f3919" containerName="maas-api" Jul 08 18:08:18.076578 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.076531 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="66400d6c-7ec8-4170-8a8e-fa42352f3919" containerName="maas-api" Jul 08 18:08:18.076616 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.076587 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="66400d6c-7ec8-4170-8a8e-fa42352f3919" containerName="maas-api" Jul 08 18:08:18.079539 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.079523 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" Jul 08 18:08:18.084074 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.084056 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-40518b0a-serving-cert\"" Jul 08 18:08:18.102067 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.102040 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85"] Jul 08 18:08:18.142607 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.142577 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:18.180323 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.180291 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70e7e21-4c2b-488e-b417-519d336df0f5-openshift-service-ca\") pod \"maas-api-e2e-mt-40518b0a-85894b86b4-x8q85\" (UID: \"a70e7e21-4c2b-488e-b417-519d336df0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" Jul 08 18:08:18.180487 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.180333 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w6wd\" (UniqueName: \"kubernetes.io/projected/a70e7e21-4c2b-488e-b417-519d336df0f5-kube-api-access-4w6wd\") pod \"maas-api-e2e-mt-40518b0a-85894b86b4-x8q85\" (UID: \"a70e7e21-4c2b-488e-b417-519d336df0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" Jul 08 18:08:18.180487 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.180423 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a70e7e21-4c2b-488e-b417-519d336df0f5-maas-api-tls\") pod \"maas-api-e2e-mt-40518b0a-85894b86b4-x8q85\" (UID: \"a70e7e21-4c2b-488e-b417-519d336df0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" Jul 08 18:08:18.281815 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.281730 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w6wd\" (UniqueName: \"kubernetes.io/projected/a70e7e21-4c2b-488e-b417-519d336df0f5-kube-api-access-4w6wd\") pod \"maas-api-e2e-mt-40518b0a-85894b86b4-x8q85\" (UID: \"a70e7e21-4c2b-488e-b417-519d336df0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" Jul 08 18:08:18.281815 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.281799 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a70e7e21-4c2b-488e-b417-519d336df0f5-maas-api-tls\") pod \"maas-api-e2e-mt-40518b0a-85894b86b4-x8q85\" (UID: \"a70e7e21-4c2b-488e-b417-519d336df0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" Jul 08 18:08:18.281994 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.281858 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70e7e21-4c2b-488e-b417-519d336df0f5-openshift-service-ca\") pod \"maas-api-e2e-mt-40518b0a-85894b86b4-x8q85\" (UID: \"a70e7e21-4c2b-488e-b417-519d336df0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" Jul 08 18:08:18.281994 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:08:18.281931 2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-serving-cert: secret "maas-api-e2e-mt-40518b0a-serving-cert" not found Jul 08 18:08:18.282065 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:08:18.282016 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a70e7e21-4c2b-488e-b417-519d336df0f5-maas-api-tls podName:a70e7e21-4c2b-488e-b417-519d336df0f5 nodeName:}" failed. No retries permitted until 2026-07-08 18:08:18.781995031 +0000 UTC m=+1774.466092302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a70e7e21-4c2b-488e-b417-519d336df0f5-maas-api-tls") pod "maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" (UID: "a70e7e21-4c2b-488e-b417-519d336df0f5") : secret "maas-api-e2e-mt-40518b0a-serving-cert" not found Jul 08 18:08:18.282494 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.282478 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70e7e21-4c2b-488e-b417-519d336df0f5-openshift-service-ca\") pod \"maas-api-e2e-mt-40518b0a-85894b86b4-x8q85\" (UID: \"a70e7e21-4c2b-488e-b417-519d336df0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" Jul 08 18:08:18.299904 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.299881 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w6wd\" (UniqueName: \"kubernetes.io/projected/a70e7e21-4c2b-488e-b417-519d336df0f5-kube-api-access-4w6wd\") pod \"maas-api-e2e-mt-40518b0a-85894b86b4-x8q85\" (UID: \"a70e7e21-4c2b-488e-b417-519d336df0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" Jul 08 18:08:18.411535 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.411498 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:18.787176 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.787137 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a70e7e21-4c2b-488e-b417-519d336df0f5-maas-api-tls\") pod \"maas-api-e2e-mt-40518b0a-85894b86b4-x8q85\" (UID: \"a70e7e21-4c2b-488e-b417-519d336df0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" Jul 08 18:08:18.789542 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.789521 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a70e7e21-4c2b-488e-b417-519d336df0f5-maas-api-tls\") pod \"maas-api-e2e-mt-40518b0a-85894b86b4-x8q85\" (UID: \"a70e7e21-4c2b-488e-b417-519d336df0f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" Jul 08 18:08:18.896500 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.896458 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:18.989446 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:18.989410 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" Jul 08 18:08:19.001792 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:19.001755 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:19.138085 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:19.138027 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85"] Jul 08 18:08:19.140718 ip-10-0-129-226 kubenswrapper[2584]: W0708 18:08:19.140688 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70e7e21_4c2b_488e_b417_519d336df0f5.slice/crio-a44b70eb4761ed32a1ef1ef690efef2df511b9b78558e2ee2edc553eb136d610 WatchSource:0}: Error finding container a44b70eb4761ed32a1ef1ef690efef2df511b9b78558e2ee2edc553eb136d610: Status 404 returned error can't find the container with id a44b70eb4761ed32a1ef1ef690efef2df511b9b78558e2ee2edc553eb136d610 Jul 08 18:08:19.672039 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:19.672005 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" event={"ID":"a70e7e21-4c2b-488e-b417-519d336df0f5","Type":"ContainerStarted","Data":"1c216c9ca3a5d716a1421d6b1928d5fcd7781554e86f2c0a76e2163441d460ec"} Jul 08 18:08:19.672218 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:19.672065 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" event={"ID":"a70e7e21-4c2b-488e-b417-519d336df0f5","Type":"ContainerStarted","Data":"a44b70eb4761ed32a1ef1ef690efef2df511b9b78558e2ee2edc553eb136d610"} Jul 08 18:08:19.672218 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:19.672159 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" Jul 08 18:08:19.708635 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:19.708583 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" podStartSLOduration=1.3692486289999999 podStartE2EDuration="1.708567385s" totalImagesPullingTime="339.318756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:08:19.706305092 +0000 UTC m=+1775.390402370" watchObservedRunningTime="2026-07-08 18:08:19.708567385 +0000 UTC m=+1775.392664661" Jul 08 18:08:22.406385 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:22.406350 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:25.683746 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:25.683710 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" Jul 08 18:08:37.366037 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:37.365956 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:08:44.976511 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:44.976478 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/1.log" Jul 08 18:08:44.979850 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:08:44.979824 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/1.log" Jul 08 18:09:05.417430 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:05.417391 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:09:05.916577 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:05.916544 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85"] Jul 08 18:09:05.916767 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:05.916741 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" podUID="a70e7e21-4c2b-488e-b417-519d336df0f5" containerName="maas-api" containerID="cri-o://1c216c9ca3a5d716a1421d6b1928d5fcd7781554e86f2c0a76e2163441d460ec" gracePeriod=30 Jul 08 18:09:05.944569 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:05.944528 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:09:06.170411 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:06.170342 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" Jul 08 18:09:06.181159 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:06.181129 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70e7e21-4c2b-488e-b417-519d336df0f5-openshift-service-ca\") pod \"a70e7e21-4c2b-488e-b417-519d336df0f5\" (UID: \"a70e7e21-4c2b-488e-b417-519d336df0f5\") " Jul 08 18:09:06.181347 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:06.181168 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4w6wd\" (UniqueName: \"kubernetes.io/projected/a70e7e21-4c2b-488e-b417-519d336df0f5-kube-api-access-4w6wd\") pod \"a70e7e21-4c2b-488e-b417-519d336df0f5\" (UID: \"a70e7e21-4c2b-488e-b417-519d336df0f5\") " Jul 08 18:09:06.181347 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:06.181222 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a70e7e21-4c2b-488e-b417-519d336df0f5-maas-api-tls\") pod \"a70e7e21-4c2b-488e-b417-519d336df0f5\" (UID: \"a70e7e21-4c2b-488e-b417-519d336df0f5\") " Jul 08 18:09:06.181578 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:06.181555 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70e7e21-4c2b-488e-b417-519d336df0f5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a70e7e21-4c2b-488e-b417-519d336df0f5" (UID: "a70e7e21-4c2b-488e-b417-519d336df0f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:09:06.183338 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:06.183311 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a70e7e21-4c2b-488e-b417-519d336df0f5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a70e7e21-4c2b-488e-b417-519d336df0f5" (UID: "a70e7e21-4c2b-488e-b417-519d336df0f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:09:06.183433 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:06.183396 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70e7e21-4c2b-488e-b417-519d336df0f5-kube-api-access-4w6wd" (OuterVolumeSpecName: "kube-api-access-4w6wd") pod "a70e7e21-4c2b-488e-b417-519d336df0f5" (UID: "a70e7e21-4c2b-488e-b417-519d336df0f5"). InnerVolumeSpecName "kube-api-access-4w6wd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:09:06.282360 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:06.282327 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70e7e21-4c2b-488e-b417-519d336df0f5-openshift-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:09:06.282360 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:06.282353 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w6wd\" (UniqueName: \"kubernetes.io/projected/a70e7e21-4c2b-488e-b417-519d336df0f5-kube-api-access-4w6wd\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:09:06.282360 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:06.282363 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a70e7e21-4c2b-488e-b417-519d336df0f5-maas-api-tls\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:09:06.944461 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:06.944417 2584 generic.go:356] "Generic (PLEG): container finished" podID="a70e7e21-4c2b-488e-b417-519d336df0f5" containerID="1c216c9ca3a5d716a1421d6b1928d5fcd7781554e86f2c0a76e2163441d460ec" exitCode=0 Jul 08 18:09:06.944461 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:06.944468 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" Jul 08 18:09:06.945005 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:06.944503 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" event={"ID":"a70e7e21-4c2b-488e-b417-519d336df0f5","Type":"ContainerDied","Data":"1c216c9ca3a5d716a1421d6b1928d5fcd7781554e86f2c0a76e2163441d460ec"} Jul 08 18:09:06.945005 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:06.944585 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85" event={"ID":"a70e7e21-4c2b-488e-b417-519d336df0f5","Type":"ContainerDied","Data":"a44b70eb4761ed32a1ef1ef690efef2df511b9b78558e2ee2edc553eb136d610"} Jul 08 18:09:06.945005 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:06.944613 2584 scope.go:117] "RemoveContainer" containerID="1c216c9ca3a5d716a1421d6b1928d5fcd7781554e86f2c0a76e2163441d460ec" Jul 08 18:09:06.956570 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:06.956535 2584 scope.go:117] "RemoveContainer" containerID="1c216c9ca3a5d716a1421d6b1928d5fcd7781554e86f2c0a76e2163441d460ec" Jul 08 18:09:06.959080 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:09:06.959048 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c216c9ca3a5d716a1421d6b1928d5fcd7781554e86f2c0a76e2163441d460ec\": container with ID starting with 1c216c9ca3a5d716a1421d6b1928d5fcd7781554e86f2c0a76e2163441d460ec not found: ID does not exist" containerID="1c216c9ca3a5d716a1421d6b1928d5fcd7781554e86f2c0a76e2163441d460ec" Jul 08 18:09:06.959196 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:06.959087 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c216c9ca3a5d716a1421d6b1928d5fcd7781554e86f2c0a76e2163441d460ec"} err="failed to get container status \"1c216c9ca3a5d716a1421d6b1928d5fcd7781554e86f2c0a76e2163441d460ec\": rpc error: code = NotFound desc = could not find container \"1c216c9ca3a5d716a1421d6b1928d5fcd7781554e86f2c0a76e2163441d460ec\": container with ID starting with 1c216c9ca3a5d716a1421d6b1928d5fcd7781554e86f2c0a76e2163441d460ec not found: ID does not exist" Jul 08 18:09:06.980474 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:06.980444 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85"] Jul 08 18:09:06.987051 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:06.987019 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-40518b0a-85894b86b4-x8q85"] Jul 08 18:09:08.958232 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:08.958197 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a70e7e21-4c2b-488e-b417-519d336df0f5" path="/var/lib/kubelet/pods/a70e7e21-4c2b-488e-b417-519d336df0f5/volumes" Jul 08 18:09:11.504669 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:11.504630 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:09:16.580361 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:16.580317 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:09:27.761623 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:27.761577 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:09:53.453612 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:53.453565 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:09:55.293755 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:09:55.293710 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:02.337830 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:10:02.337792 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:02.441776 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:10:02.441734 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:02.560176 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:10:02.560142 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:03.863757 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:10:03.863709 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:03.943405 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:10:03.943368 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:04.040445 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:10:04.040402 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:04.888085 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:10:04.888037 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:04.981207 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:10:04.981167 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:05.081344 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:10:05.081307 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:06.698830 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:10:06.698788 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:14.020560 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:10:14.020479 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:14.120913 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:10:14.120882 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:14.807362 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:10:14.807311 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:22.602381 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:10:22.602342 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:22.704034 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:10:22.703989 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:22.803485 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:10:22.803431 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:33.445192 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:10:33.445158 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:33.817542 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:10:33.817459 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:10:54.833051 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:10:54.833012 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:01.718359 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:01.718314 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:01.817363 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:01.817333 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:02.123191 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:02.123152 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:02.522141 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:02.522040 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:02.868339 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:02.868305 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:05.422164 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.422130 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:05.513084 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.513044 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:05.546435 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.546401 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c"] Jul 08 18:11:05.546828 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.546816 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a70e7e21-4c2b-488e-b417-519d336df0f5" containerName="maas-api" Jul 08 18:11:05.546882 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.546830 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a70e7e21-4c2b-488e-b417-519d336df0f5" containerName="maas-api" Jul 08 18:11:05.546921 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.546892 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a70e7e21-4c2b-488e-b417-519d336df0f5" containerName="maas-api" Jul 08 18:11:05.550133 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.550112 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" Jul 08 18:11:05.554345 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.554308 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-bd74c9-serving-cert\"" Jul 08 18:11:05.574118 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.574085 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c"] Jul 08 18:11:05.606780 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.606730 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:05.623944 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.623906 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a03af321-42ee-4166-b493-e7448a7d2281-maas-api-tls\") pod \"maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c\" (UID: \"a03af321-42ee-4166-b493-e7448a7d2281\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" Jul 08 18:11:05.624129 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.624004 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnpg7\" (UniqueName: \"kubernetes.io/projected/a03af321-42ee-4166-b493-e7448a7d2281-kube-api-access-lnpg7\") pod \"maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c\" (UID: \"a03af321-42ee-4166-b493-e7448a7d2281\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" Jul 08 18:11:05.624129 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.624046 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a03af321-42ee-4166-b493-e7448a7d2281-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c\" (UID: \"a03af321-42ee-4166-b493-e7448a7d2281\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" Jul 08 18:11:05.725132 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.725025 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a03af321-42ee-4166-b493-e7448a7d2281-maas-api-tls\") pod \"maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c\" (UID: \"a03af321-42ee-4166-b493-e7448a7d2281\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" Jul 08 18:11:05.725132 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.725113 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnpg7\" (UniqueName: \"kubernetes.io/projected/a03af321-42ee-4166-b493-e7448a7d2281-kube-api-access-lnpg7\") pod \"maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c\" (UID: \"a03af321-42ee-4166-b493-e7448a7d2281\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" Jul 08 18:11:05.725381 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.725143 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a03af321-42ee-4166-b493-e7448a7d2281-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c\" (UID: \"a03af321-42ee-4166-b493-e7448a7d2281\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" Jul 08 18:11:05.725924 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.725899 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a03af321-42ee-4166-b493-e7448a7d2281-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c\" (UID: \"a03af321-42ee-4166-b493-e7448a7d2281\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" Jul 08 18:11:05.727582 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.727561 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a03af321-42ee-4166-b493-e7448a7d2281-maas-api-tls\") pod \"maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c\" (UID: \"a03af321-42ee-4166-b493-e7448a7d2281\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" Jul 08 18:11:05.739495 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.739462 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnpg7\" (UniqueName: \"kubernetes.io/projected/a03af321-42ee-4166-b493-e7448a7d2281-kube-api-access-lnpg7\") pod \"maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c\" (UID: \"a03af321-42ee-4166-b493-e7448a7d2281\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" Jul 08 18:11:05.861291 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:05.861229 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" Jul 08 18:11:06.016304 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:06.016272 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c"] Jul 08 18:11:06.017726 ip-10-0-129-226 kubenswrapper[2584]: W0708 18:11:06.017700 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03af321_42ee_4166_b493_e7448a7d2281.slice/crio-ba8da342a8aaed3c64115e928af323931c0b787f03ffdc0c110080aeb6da81eb WatchSource:0}: Error finding container ba8da342a8aaed3c64115e928af323931c0b787f03ffdc0c110080aeb6da81eb: Status 404 returned error can't find the container with id ba8da342a8aaed3c64115e928af323931c0b787f03ffdc0c110080aeb6da81eb Jul 08 18:11:06.019406 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:06.019389 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 18:11:06.458170 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:06.458140 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:06.549538 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:06.549491 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:06.601681 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:06.601651 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" event={"ID":"a03af321-42ee-4166-b493-e7448a7d2281","Type":"ContainerStarted","Data":"ba8da342a8aaed3c64115e928af323931c0b787f03ffdc0c110080aeb6da81eb"} Jul 08 18:11:07.609086 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:07.609048 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" event={"ID":"a03af321-42ee-4166-b493-e7448a7d2281","Type":"ContainerStarted","Data":"d47eb7ac49ecadbe00c750c95b87633cb160509c3912252b98cd19db322d7bce"} Jul 08 18:11:07.609569 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:07.609116 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" Jul 08 18:11:07.648475 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:07.648416 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" podStartSLOduration=1.827422712 podStartE2EDuration="2.648399186s" totalImagesPullingTime="820.976474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:11:07.647128643 +0000 UTC m=+1943.331225920" watchObservedRunningTime="2026-07-08 18:11:07.648399186 +0000 UTC m=+1943.332496463" Jul 08 18:11:09.960825 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:09.960782 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:11.158604 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:11.158566 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:11.249268 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:11.249202 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:13.448572 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:13.448528 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:13.547827 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:13.547787 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7"] Jul 08 18:11:13.552398 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:13.552369 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" Jul 08 18:11:13.554863 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:13.554838 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-97d4b9-serving-cert\"" Jul 08 18:11:13.563002 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:13.562975 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7"] Jul 08 18:11:13.571380 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:13.571349 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:13.601549 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:13.601509 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx57w\" (UniqueName: \"kubernetes.io/projected/ab7b7a80-5043-403c-8f3c-c12e66933c8a-kube-api-access-nx57w\") pod \"maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7\" (UID: \"ab7b7a80-5043-403c-8f3c-c12e66933c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" Jul 08 18:11:13.601710 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:13.601579 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7b7a80-5043-403c-8f3c-c12e66933c8a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7\" (UID: \"ab7b7a80-5043-403c-8f3c-c12e66933c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" Jul 08 18:11:13.601710 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:13.601655 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7b7a80-5043-403c-8f3c-c12e66933c8a-maas-api-tls\") pod \"maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7\" (UID: \"ab7b7a80-5043-403c-8f3c-c12e66933c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" Jul 08 18:11:13.621339 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:13.621310 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" Jul 08 18:11:13.687814 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:13.687775 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:13.702782 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:13.702719 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7b7a80-5043-403c-8f3c-c12e66933c8a-maas-api-tls\") pod \"maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7\" (UID: \"ab7b7a80-5043-403c-8f3c-c12e66933c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" Jul 08 18:11:13.702782 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:13.702779 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx57w\" (UniqueName: \"kubernetes.io/projected/ab7b7a80-5043-403c-8f3c-c12e66933c8a-kube-api-access-nx57w\") pod \"maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7\" (UID: \"ab7b7a80-5043-403c-8f3c-c12e66933c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" Jul 08 18:11:13.702946 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:13.702825 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7b7a80-5043-403c-8f3c-c12e66933c8a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7\" (UID: \"ab7b7a80-5043-403c-8f3c-c12e66933c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" Jul 08 18:11:13.703551 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:13.703524 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7b7a80-5043-403c-8f3c-c12e66933c8a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7\" (UID: \"ab7b7a80-5043-403c-8f3c-c12e66933c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" Jul 08 18:11:13.705167 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:13.705145 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7b7a80-5043-403c-8f3c-c12e66933c8a-maas-api-tls\") pod \"maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7\" (UID: \"ab7b7a80-5043-403c-8f3c-c12e66933c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" Jul 08 18:11:13.720145 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:13.720122 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx57w\" (UniqueName: \"kubernetes.io/projected/ab7b7a80-5043-403c-8f3c-c12e66933c8a-kube-api-access-nx57w\") pod \"maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7\" (UID: \"ab7b7a80-5043-403c-8f3c-c12e66933c8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" Jul 08 18:11:13.863804 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:13.863771 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" Jul 08 18:11:14.024230 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:14.024193 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7"] Jul 08 18:11:14.024739 ip-10-0-129-226 kubenswrapper[2584]: W0708 18:11:14.024714 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7b7a80_5043_403c_8f3c_c12e66933c8a.slice/crio-4924981c14167921548cb3916797d12d6043768409b22b958356eb126feb5dcf WatchSource:0}: Error finding container 4924981c14167921548cb3916797d12d6043768409b22b958356eb126feb5dcf: Status 404 returned error can't find the container with id 4924981c14167921548cb3916797d12d6043768409b22b958356eb126feb5dcf Jul 08 18:11:14.417852 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:14.417805 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:14.517732 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:14.517696 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:14.652047 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:14.652012 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" event={"ID":"ab7b7a80-5043-403c-8f3c-c12e66933c8a","Type":"ContainerStarted","Data":"4924981c14167921548cb3916797d12d6043768409b22b958356eb126feb5dcf"} Jul 08 18:11:15.660579 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:15.660528 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" event={"ID":"ab7b7a80-5043-403c-8f3c-c12e66933c8a","Type":"ContainerStarted","Data":"2ee343fc5a9472acd9f1a09164eff197b15850069bf7b60bcab75b156f8d3d9f"} Jul 08 18:11:15.661047 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:15.660717 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" Jul 08 18:11:15.684973 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:15.684918 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" podStartSLOduration=1.8975689569999998 podStartE2EDuration="2.684902172s" totalImagesPullingTime="787.333215ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:11:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:11:15.682425669 +0000 UTC m=+1951.366522946" watchObservedRunningTime="2026-07-08 18:11:15.684902172 +0000 UTC m=+1951.368999448" Jul 08 18:11:17.998909 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:17.998872 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:21.672576 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:21.672545 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" Jul 08 18:11:24.233237 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:24.233183 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:24.738309 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:24.738276 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:28.071124 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:28.071080 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:28.168330 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:28.168284 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:36.551188 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:36.551156 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:36.651545 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:36.651492 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:39.246254 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:39.246152 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:39.348201 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:39.348153 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:11:53.442914 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:11:53.442870 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:12:32.311286 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:32.311229 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:12:33.228322 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:33.228264 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:12:33.406339 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:33.406300 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:12:33.768913 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:33.768875 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:12:33.977455 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:33.977422 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:12:34.084194 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:34.084112 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:12:34.372749 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:34.372717 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c"] Jul 08 18:12:34.372948 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:34.372925 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" podUID="a03af321-42ee-4166-b493-e7448a7d2281" containerName="maas-api" containerID="cri-o://d47eb7ac49ecadbe00c750c95b87633cb160509c3912252b98cd19db322d7bce" gracePeriod=30 Jul 08 18:12:34.401820 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:34.401788 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:12:34.624279 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:34.624207 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" Jul 08 18:12:34.702105 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:34.702071 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a03af321-42ee-4166-b493-e7448a7d2281-maas-api-tls\") pod \"a03af321-42ee-4166-b493-e7448a7d2281\" (UID: \"a03af321-42ee-4166-b493-e7448a7d2281\") " Jul 08 18:12:34.702306 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:34.702115 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a03af321-42ee-4166-b493-e7448a7d2281-openshift-service-ca\") pod \"a03af321-42ee-4166-b493-e7448a7d2281\" (UID: \"a03af321-42ee-4166-b493-e7448a7d2281\") " Jul 08 18:12:34.702306 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:34.702137 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnpg7\" (UniqueName: \"kubernetes.io/projected/a03af321-42ee-4166-b493-e7448a7d2281-kube-api-access-lnpg7\") pod \"a03af321-42ee-4166-b493-e7448a7d2281\" (UID: \"a03af321-42ee-4166-b493-e7448a7d2281\") " Jul 08 18:12:34.702519 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:34.702495 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a03af321-42ee-4166-b493-e7448a7d2281-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a03af321-42ee-4166-b493-e7448a7d2281" (UID: "a03af321-42ee-4166-b493-e7448a7d2281"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:12:34.704340 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:34.704313 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a03af321-42ee-4166-b493-e7448a7d2281-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a03af321-42ee-4166-b493-e7448a7d2281" (UID: "a03af321-42ee-4166-b493-e7448a7d2281"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:12:34.704460 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:34.704440 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03af321-42ee-4166-b493-e7448a7d2281-kube-api-access-lnpg7" (OuterVolumeSpecName: "kube-api-access-lnpg7") pod "a03af321-42ee-4166-b493-e7448a7d2281" (UID: "a03af321-42ee-4166-b493-e7448a7d2281"). InnerVolumeSpecName "kube-api-access-lnpg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:12:34.803771 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:34.803732 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a03af321-42ee-4166-b493-e7448a7d2281-maas-api-tls\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:12:34.803771 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:34.803763 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a03af321-42ee-4166-b493-e7448a7d2281-openshift-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:12:34.803771 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:34.803772 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnpg7\" (UniqueName: \"kubernetes.io/projected/a03af321-42ee-4166-b493-e7448a7d2281-kube-api-access-lnpg7\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:12:35.113461 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:35.113431 2584 generic.go:356] "Generic (PLEG): container finished" podID="a03af321-42ee-4166-b493-e7448a7d2281" containerID="d47eb7ac49ecadbe00c750c95b87633cb160509c3912252b98cd19db322d7bce" exitCode=0 Jul 08 18:12:35.113638 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:35.113477 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" Jul 08 18:12:35.113638 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:35.113522 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" event={"ID":"a03af321-42ee-4166-b493-e7448a7d2281","Type":"ContainerDied","Data":"d47eb7ac49ecadbe00c750c95b87633cb160509c3912252b98cd19db322d7bce"} Jul 08 18:12:35.113638 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:35.113569 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c" event={"ID":"a03af321-42ee-4166-b493-e7448a7d2281","Type":"ContainerDied","Data":"ba8da342a8aaed3c64115e928af323931c0b787f03ffdc0c110080aeb6da81eb"} Jul 08 18:12:35.113638 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:35.113591 2584 scope.go:117] "RemoveContainer" containerID="d47eb7ac49ecadbe00c750c95b87633cb160509c3912252b98cd19db322d7bce" Jul 08 18:12:35.122917 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:35.122896 2584 scope.go:117] "RemoveContainer" containerID="d47eb7ac49ecadbe00c750c95b87633cb160509c3912252b98cd19db322d7bce" Jul 08 18:12:35.123215 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:12:35.123195 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47eb7ac49ecadbe00c750c95b87633cb160509c3912252b98cd19db322d7bce\": container with ID starting with d47eb7ac49ecadbe00c750c95b87633cb160509c3912252b98cd19db322d7bce not found: ID does not exist" containerID="d47eb7ac49ecadbe00c750c95b87633cb160509c3912252b98cd19db322d7bce" Jul 08 18:12:35.123296 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:35.123222 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47eb7ac49ecadbe00c750c95b87633cb160509c3912252b98cd19db322d7bce"} err="failed to get container status \"d47eb7ac49ecadbe00c750c95b87633cb160509c3912252b98cd19db322d7bce\": rpc error: code = NotFound desc = could not find container \"d47eb7ac49ecadbe00c750c95b87633cb160509c3912252b98cd19db322d7bce\": container with ID starting with d47eb7ac49ecadbe00c750c95b87633cb160509c3912252b98cd19db322d7bce not found: ID does not exist" Jul 08 18:12:35.136381 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:35.136353 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c"] Jul 08 18:12:35.148176 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:35.148150 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bd74c9-577644dd4-8nk9c"] Jul 08 18:12:36.957029 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:36.956995 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a03af321-42ee-4166-b493-e7448a7d2281" path="/var/lib/kubelet/pods/a03af321-42ee-4166-b493-e7448a7d2281/volumes" Jul 08 18:12:38.839558 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:38.839520 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:12:44.102432 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:44.102396 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:12:57.800872 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:57.800827 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:12:57.903077 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:12:57.903038 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:16.962098 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:16.962004 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:17.233756 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:17.233660 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:17.921570 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:17.921523 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:18.319576 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:18.319480 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:18.491953 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:18.491906 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:18.805934 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:18.805892 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:18.807901 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:18.807876 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7"] Jul 08 18:13:18.808196 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:18.808164 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" podUID="ab7b7a80-5043-403c-8f3c-c12e66933c8a" containerName="maas-api" containerID="cri-o://2ee343fc5a9472acd9f1a09164eff197b15850069bf7b60bcab75b156f8d3d9f" gracePeriod=30 Jul 08 18:13:18.899149 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:18.899109 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:20.060371 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.060344 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" Jul 08 18:13:20.100605 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.100575 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7b7a80-5043-403c-8f3c-c12e66933c8a-openshift-service-ca\") pod \"ab7b7a80-5043-403c-8f3c-c12e66933c8a\" (UID: \"ab7b7a80-5043-403c-8f3c-c12e66933c8a\") " Jul 08 18:13:20.100762 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.100621 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7b7a80-5043-403c-8f3c-c12e66933c8a-maas-api-tls\") pod \"ab7b7a80-5043-403c-8f3c-c12e66933c8a\" (UID: \"ab7b7a80-5043-403c-8f3c-c12e66933c8a\") " Jul 08 18:13:20.100762 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.100644 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nx57w\" (UniqueName: \"kubernetes.io/projected/ab7b7a80-5043-403c-8f3c-c12e66933c8a-kube-api-access-nx57w\") pod \"ab7b7a80-5043-403c-8f3c-c12e66933c8a\" (UID: \"ab7b7a80-5043-403c-8f3c-c12e66933c8a\") " Jul 08 18:13:20.101056 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.101022 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7b7a80-5043-403c-8f3c-c12e66933c8a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ab7b7a80-5043-403c-8f3c-c12e66933c8a" (UID: "ab7b7a80-5043-403c-8f3c-c12e66933c8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:13:20.102801 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.102768 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab7b7a80-5043-403c-8f3c-c12e66933c8a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ab7b7a80-5043-403c-8f3c-c12e66933c8a" (UID: "ab7b7a80-5043-403c-8f3c-c12e66933c8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:13:20.102908 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.102806 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7b7a80-5043-403c-8f3c-c12e66933c8a-kube-api-access-nx57w" (OuterVolumeSpecName: "kube-api-access-nx57w") pod "ab7b7a80-5043-403c-8f3c-c12e66933c8a" (UID: "ab7b7a80-5043-403c-8f3c-c12e66933c8a"). InnerVolumeSpecName "kube-api-access-nx57w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:13:20.201826 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.201779 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7b7a80-5043-403c-8f3c-c12e66933c8a-openshift-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:13:20.201826 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.201821 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab7b7a80-5043-403c-8f3c-c12e66933c8a-maas-api-tls\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:13:20.201826 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.201831 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx57w\" (UniqueName: \"kubernetes.io/projected/ab7b7a80-5043-403c-8f3c-c12e66933c8a-kube-api-access-nx57w\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:13:20.363722 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.363635 2584 generic.go:356] "Generic (PLEG): container finished" podID="ab7b7a80-5043-403c-8f3c-c12e66933c8a" containerID="2ee343fc5a9472acd9f1a09164eff197b15850069bf7b60bcab75b156f8d3d9f" exitCode=0 Jul 08 18:13:20.363722 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.363684 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" Jul 08 18:13:20.363722 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.363707 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" event={"ID":"ab7b7a80-5043-403c-8f3c-c12e66933c8a","Type":"ContainerDied","Data":"2ee343fc5a9472acd9f1a09164eff197b15850069bf7b60bcab75b156f8d3d9f"} Jul 08 18:13:20.363958 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.363747 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7" event={"ID":"ab7b7a80-5043-403c-8f3c-c12e66933c8a","Type":"ContainerDied","Data":"4924981c14167921548cb3916797d12d6043768409b22b958356eb126feb5dcf"} Jul 08 18:13:20.363958 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.363770 2584 scope.go:117] "RemoveContainer" containerID="2ee343fc5a9472acd9f1a09164eff197b15850069bf7b60bcab75b156f8d3d9f" Jul 08 18:13:20.375288 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.375260 2584 scope.go:117] "RemoveContainer" containerID="2ee343fc5a9472acd9f1a09164eff197b15850069bf7b60bcab75b156f8d3d9f" Jul 08 18:13:20.375825 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:13:20.375799 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee343fc5a9472acd9f1a09164eff197b15850069bf7b60bcab75b156f8d3d9f\": container with ID starting with 2ee343fc5a9472acd9f1a09164eff197b15850069bf7b60bcab75b156f8d3d9f not found: ID does not exist" containerID="2ee343fc5a9472acd9f1a09164eff197b15850069bf7b60bcab75b156f8d3d9f" Jul 08 18:13:20.375941 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.375834 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee343fc5a9472acd9f1a09164eff197b15850069bf7b60bcab75b156f8d3d9f"} err="failed to get container status \"2ee343fc5a9472acd9f1a09164eff197b15850069bf7b60bcab75b156f8d3d9f\": rpc error: code = NotFound desc = could not find container \"2ee343fc5a9472acd9f1a09164eff197b15850069bf7b60bcab75b156f8d3d9f\": container with ID starting with 2ee343fc5a9472acd9f1a09164eff197b15850069bf7b60bcab75b156f8d3d9f not found: ID does not exist" Jul 08 18:13:20.391832 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.391803 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7"] Jul 08 18:13:20.397532 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.397507 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-97d4b9-b9f88d4fc-bncp7"] Jul 08 18:13:20.957232 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:20.957198 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab7b7a80-5043-403c-8f3c-c12e66933c8a" path="/var/lib/kubelet/pods/ab7b7a80-5043-403c-8f3c-c12e66933c8a/volumes" Jul 08 18:13:24.954523 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:24.954482 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:25.039044 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:25.039007 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:25.099601 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:25.099557 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:25.648073 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:25.648021 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:25.965704 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:25.965622 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4"] Jul 08 18:13:25.966304 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:25.966281 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab7b7a80-5043-403c-8f3c-c12e66933c8a" containerName="maas-api" Jul 08 18:13:25.966410 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:25.966307 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab7b7a80-5043-403c-8f3c-c12e66933c8a" containerName="maas-api" Jul 08 18:13:25.966410 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:25.966334 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a03af321-42ee-4166-b493-e7448a7d2281" containerName="maas-api" Jul 08 18:13:25.966410 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:25.966342 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="a03af321-42ee-4166-b493-e7448a7d2281" containerName="maas-api" Jul 08 18:13:25.966561 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:25.966486 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab7b7a80-5043-403c-8f3c-c12e66933c8a" containerName="maas-api" Jul 08 18:13:25.966561 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:25.966501 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="a03af321-42ee-4166-b493-e7448a7d2281" containerName="maas-api" Jul 08 18:13:25.970901 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:25.970877 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:25.973444 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:25.973421 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 18:13:25.973444 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:25.973421 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-988034-openshift-default-dockercfg-j5qb7\"" Jul 08 18:13:25.981668 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:25.981643 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4"] Jul 08 18:13:25.984309 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:25.984277 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:26.054376 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.054343 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhr84\" (UniqueName: \"kubernetes.io/projected/f4635485-8278-4278-988b-4b889ddb42eb-kube-api-access-vhr84\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.054615 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.054391 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-istio-envoy\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.054615 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.054551 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4635485-8278-4278-988b-4b889ddb42eb-istio-token\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.054736 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.054626 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-credential-socket\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.054736 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.054665 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-workload-socket\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.054824 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.054735 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4635485-8278-4278-988b-4b889ddb42eb-istiod-ca-cert\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.054824 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.054770 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-istio-data\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.054824 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.054804 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-workload-certs\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.054824 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.054821 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4635485-8278-4278-988b-4b889ddb42eb-istio-podinfo\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.156335 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.156299 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4635485-8278-4278-988b-4b889ddb42eb-istiod-ca-cert\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.156335 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.156341 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-istio-data\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.156575 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.156365 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-workload-certs\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.156575 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.156388 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4635485-8278-4278-988b-4b889ddb42eb-istio-podinfo\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.156575 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.156433 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhr84\" (UniqueName: \"kubernetes.io/projected/f4635485-8278-4278-988b-4b889ddb42eb-kube-api-access-vhr84\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.156575 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.156464 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-istio-envoy\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.156575 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.156555 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4635485-8278-4278-988b-4b889ddb42eb-istio-token\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.156849 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.156630 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-credential-socket\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.156849 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.156672 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-workload-socket\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.156849 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.156783 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-workload-certs\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.157017 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.156884 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-istio-data\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.157017 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.156972 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-workload-socket\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.157128 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.157067 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-credential-socket\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.157184 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.157141 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4635485-8278-4278-988b-4b889ddb42eb-istiod-ca-cert\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.158662 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.158629 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-istio-envoy\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.158886 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.158859 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4635485-8278-4278-988b-4b889ddb42eb-istio-podinfo\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.160153 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.160131 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:26.165329 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.165310 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhr84\" (UniqueName: \"kubernetes.io/projected/f4635485-8278-4278-988b-4b889ddb42eb-kube-api-access-vhr84\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.165528 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.165511 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4635485-8278-4278-988b-4b889ddb42eb-istio-token\") pod \"e2e-api-a-988034-openshift-default-5949bbb7-h8lc4\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.282485 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.282383 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:26.427790 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:26.427755 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4"] Jul 08 18:13:26.428176 ip-10-0-129-226 kubenswrapper[2584]: W0708 18:13:26.428151 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4635485_8278_4278_988b_4b889ddb42eb.slice/crio-fdd2e30869c3397cff9ab54a6e188f806ca637ee03f43ada32cabd1c6dc44b22 WatchSource:0}: Error finding container fdd2e30869c3397cff9ab54a6e188f806ca637ee03f43ada32cabd1c6dc44b22: Status 404 returned error can't find the container with id fdd2e30869c3397cff9ab54a6e188f806ca637ee03f43ada32cabd1c6dc44b22 Jul 08 18:13:27.406571 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:27.406528 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" event={"ID":"f4635485-8278-4278-988b-4b889ddb42eb","Type":"ContainerStarted","Data":"fdd2e30869c3397cff9ab54a6e188f806ca637ee03f43ada32cabd1c6dc44b22"} Jul 08 18:13:29.070007 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:29.069966 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 18:13:29.070396 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:29.070055 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 18:13:29.070396 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:29.070100 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 18:13:29.420538 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:29.420504 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" event={"ID":"f4635485-8278-4278-988b-4b889ddb42eb","Type":"ContainerStarted","Data":"c741da0cc2cdbd38702babe96719f8d6ac40e7591602419f64cd96ac9d83e21f"} Jul 08 18:13:29.463181 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:29.463123 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" podStartSLOduration=1.824077444 podStartE2EDuration="4.463105449s" totalImagesPullingTime="2.639028005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:13:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:13:29.461450747 +0000 UTC m=+2085.145548023" watchObservedRunningTime="2026-07-08 18:13:29.463105449 +0000 UTC m=+2085.147203138" Jul 08 18:13:30.283177 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:30.283137 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:30.288038 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:30.288012 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:30.426255 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:30.426224 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:30.427363 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:30.427339 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:31.457623 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:31.457574 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:31.553173 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:31.553143 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:31.554801 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:31.554779 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc"] Jul 08 18:13:31.558707 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:31.558686 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" Jul 08 18:13:31.561141 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:31.561121 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-988034-serving-cert\"" Jul 08 18:13:31.569336 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:31.569308 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc"] Jul 08 18:13:31.620357 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:31.620323 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:31.707539 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:31.707501 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-openshift-service-ca\") pod \"maas-api-e2e-api-a-988034-9c57ff54-b6wkc\" (UID: \"ed9fefee-0d6e-4804-9ba5-7c720cd7601b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" Jul 08 18:13:31.707734 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:31.707569 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-maas-api-tls\") pod \"maas-api-e2e-api-a-988034-9c57ff54-b6wkc\" (UID: \"ed9fefee-0d6e-4804-9ba5-7c720cd7601b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" Jul 08 18:13:31.707734 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:31.707610 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbv2c\" (UniqueName: \"kubernetes.io/projected/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-kube-api-access-gbv2c\") pod \"maas-api-e2e-api-a-988034-9c57ff54-b6wkc\" (UID: \"ed9fefee-0d6e-4804-9ba5-7c720cd7601b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" Jul 08 18:13:31.808971 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:31.808925 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-openshift-service-ca\") pod \"maas-api-e2e-api-a-988034-9c57ff54-b6wkc\" (UID: \"ed9fefee-0d6e-4804-9ba5-7c720cd7601b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" Jul 08 18:13:31.809176 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:31.809024 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-maas-api-tls\") pod \"maas-api-e2e-api-a-988034-9c57ff54-b6wkc\" (UID: \"ed9fefee-0d6e-4804-9ba5-7c720cd7601b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" Jul 08 18:13:31.809176 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:31.809075 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbv2c\" (UniqueName: \"kubernetes.io/projected/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-kube-api-access-gbv2c\") pod \"maas-api-e2e-api-a-988034-9c57ff54-b6wkc\" (UID: \"ed9fefee-0d6e-4804-9ba5-7c720cd7601b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" Jul 08 18:13:31.809357 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:13:31.809168 2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-988034-serving-cert: secret "maas-api-e2e-api-a-988034-serving-cert" not found Jul 08 18:13:31.809357 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:13:31.809285 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-maas-api-tls podName:ed9fefee-0d6e-4804-9ba5-7c720cd7601b nodeName:}" failed. No retries permitted until 2026-07-08 18:13:32.309235057 +0000 UTC m=+2087.993332313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-maas-api-tls") pod "maas-api-e2e-api-a-988034-9c57ff54-b6wkc" (UID: "ed9fefee-0d6e-4804-9ba5-7c720cd7601b") : secret "maas-api-e2e-api-a-988034-serving-cert" not found Jul 08 18:13:31.809679 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:31.809660 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-openshift-service-ca\") pod \"maas-api-e2e-api-a-988034-9c57ff54-b6wkc\" (UID: \"ed9fefee-0d6e-4804-9ba5-7c720cd7601b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" Jul 08 18:13:31.821299 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:31.821273 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbv2c\" (UniqueName: \"kubernetes.io/projected/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-kube-api-access-gbv2c\") pod \"maas-api-e2e-api-a-988034-9c57ff54-b6wkc\" (UID: \"ed9fefee-0d6e-4804-9ba5-7c720cd7601b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" Jul 08 18:13:31.996140 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:31.996050 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:32.186648 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:32.186611 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:32.288285 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:32.288185 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:32.314370 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:32.314337 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-maas-api-tls\") pod \"maas-api-e2e-api-a-988034-9c57ff54-b6wkc\" (UID: \"ed9fefee-0d6e-4804-9ba5-7c720cd7601b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" Jul 08 18:13:32.316730 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:32.316708 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-maas-api-tls\") pod \"maas-api-e2e-api-a-988034-9c57ff54-b6wkc\" (UID: \"ed9fefee-0d6e-4804-9ba5-7c720cd7601b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" Jul 08 18:13:32.469514 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:32.469474 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" Jul 08 18:13:32.621629 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:32.621603 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc"] Jul 08 18:13:32.623732 ip-10-0-129-226 kubenswrapper[2584]: W0708 18:13:32.623691 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9fefee_0d6e_4804_9ba5_7c720cd7601b.slice/crio-24b22a3d8e0f3e823f1b0e721f1d613d6ba61bd8c29fb8f7bb951e57c347ccfa WatchSource:0}: Error finding container 24b22a3d8e0f3e823f1b0e721f1d613d6ba61bd8c29fb8f7bb951e57c347ccfa: Status 404 returned error can't find the container with id 24b22a3d8e0f3e823f1b0e721f1d613d6ba61bd8c29fb8f7bb951e57c347ccfa Jul 08 18:13:33.444724 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:33.444686 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" event={"ID":"ed9fefee-0d6e-4804-9ba5-7c720cd7601b","Type":"ContainerStarted","Data":"24b22a3d8e0f3e823f1b0e721f1d613d6ba61bd8c29fb8f7bb951e57c347ccfa"} Jul 08 18:13:34.453063 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:34.453019 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" event={"ID":"ed9fefee-0d6e-4804-9ba5-7c720cd7601b","Type":"ContainerStarted","Data":"56f114c807d78cd2da39b51c1c95bdf85864b59658872f33f7c5785dc5546d4d"} Jul 08 18:13:34.453536 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:34.453092 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" Jul 08 18:13:34.474678 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:34.474625 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" podStartSLOduration=2.0406251539999998 podStartE2EDuration="3.474607081s" totalImagesPullingTime="1.433981927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:13:34.472756652 +0000 UTC m=+2090.156853931" watchObservedRunningTime="2026-07-08 18:13:34.474607081 +0000 UTC m=+2090.158704360" Jul 08 18:13:35.991717 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:35.991680 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:36.496903 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:36.496851 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:36.845279 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:36.845088 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:37.303965 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:37.303935 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:40.060847 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:40.060804 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:40.164805 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:40.164761 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:40.301448 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:40.301414 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l"] Jul 08 18:13:40.306852 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:40.306826 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" Jul 08 18:13:40.309721 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:40.309697 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1dc085-serving-cert\"" Jul 08 18:13:40.314180 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:40.314072 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l"] Jul 08 18:13:40.357705 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:40.357669 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:40.394798 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:40.394758 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6822cc38-4f9c-438f-829e-976160a5b1dc-openshift-service-ca\") pod \"maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l\" (UID: \"6822cc38-4f9c-438f-829e-976160a5b1dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" Jul 08 18:13:40.394956 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:40.394821 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdb5c\" (UniqueName: \"kubernetes.io/projected/6822cc38-4f9c-438f-829e-976160a5b1dc-kube-api-access-bdb5c\") pod \"maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l\" (UID: \"6822cc38-4f9c-438f-829e-976160a5b1dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" Jul 08 18:13:40.394956 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:40.394894 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6822cc38-4f9c-438f-829e-976160a5b1dc-maas-api-tls\") pod \"maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l\" (UID: \"6822cc38-4f9c-438f-829e-976160a5b1dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" Jul 08 18:13:40.465232 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:40.465202 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" Jul 08 18:13:40.498679 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:40.498643 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6822cc38-4f9c-438f-829e-976160a5b1dc-openshift-service-ca\") pod \"maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l\" (UID: \"6822cc38-4f9c-438f-829e-976160a5b1dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" Jul 08 18:13:40.499001 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:40.498965 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdb5c\" (UniqueName: \"kubernetes.io/projected/6822cc38-4f9c-438f-829e-976160a5b1dc-kube-api-access-bdb5c\") pod \"maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l\" (UID: \"6822cc38-4f9c-438f-829e-976160a5b1dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" Jul 08 18:13:40.499108 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:40.499087 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6822cc38-4f9c-438f-829e-976160a5b1dc-maas-api-tls\") pod \"maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l\" (UID: \"6822cc38-4f9c-438f-829e-976160a5b1dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" Jul 08 18:13:40.499434 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:13:40.499410 2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-serving-cert: secret "maas-api-e2e-api-b-1dc085-serving-cert" not found Jul 08 18:13:40.499712 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:13:40.499691 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6822cc38-4f9c-438f-829e-976160a5b1dc-maas-api-tls podName:6822cc38-4f9c-438f-829e-976160a5b1dc nodeName:}" failed. No retries permitted until 2026-07-08 18:13:40.99966733 +0000 UTC m=+2096.683764593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6822cc38-4f9c-438f-829e-976160a5b1dc-maas-api-tls") pod "maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" (UID: "6822cc38-4f9c-438f-829e-976160a5b1dc") : secret "maas-api-e2e-api-b-1dc085-serving-cert" not found Jul 08 18:13:40.499863 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:40.499824 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6822cc38-4f9c-438f-829e-976160a5b1dc-openshift-service-ca\") pod \"maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l\" (UID: \"6822cc38-4f9c-438f-829e-976160a5b1dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" Jul 08 18:13:40.516481 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:40.516457 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdb5c\" (UniqueName: \"kubernetes.io/projected/6822cc38-4f9c-438f-829e-976160a5b1dc-kube-api-access-bdb5c\") pod \"maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l\" (UID: \"6822cc38-4f9c-438f-829e-976160a5b1dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" Jul 08 18:13:40.847162 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:40.847122 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:41.003981 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:41.003944 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6822cc38-4f9c-438f-829e-976160a5b1dc-maas-api-tls\") pod \"maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l\" (UID: \"6822cc38-4f9c-438f-829e-976160a5b1dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" Jul 08 18:13:41.006645 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:41.006619 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6822cc38-4f9c-438f-829e-976160a5b1dc-maas-api-tls\") pod \"maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l\" (UID: \"6822cc38-4f9c-438f-829e-976160a5b1dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" Jul 08 18:13:41.022628 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:41.022585 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:41.121878 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:41.121842 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:41.219545 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:41.219510 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" Jul 08 18:13:41.381229 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:41.381200 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l"] Jul 08 18:13:41.383131 ip-10-0-129-226 kubenswrapper[2584]: W0708 18:13:41.383093 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6822cc38_4f9c_438f_829e_976160a5b1dc.slice/crio-837139f00471f2b9069e37b77a6e14d1743becdb4dcec32e12d1bc8a09b51eb3 WatchSource:0}: Error finding container 837139f00471f2b9069e37b77a6e14d1743becdb4dcec32e12d1bc8a09b51eb3: Status 404 returned error can't find the container with id 837139f00471f2b9069e37b77a6e14d1743becdb4dcec32e12d1bc8a09b51eb3 Jul 08 18:13:41.495799 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:41.495763 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" event={"ID":"6822cc38-4f9c-438f-829e-976160a5b1dc","Type":"ContainerStarted","Data":"837139f00471f2b9069e37b77a6e14d1743becdb4dcec32e12d1bc8a09b51eb3"} Jul 08 18:13:42.502743 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:42.502702 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" event={"ID":"6822cc38-4f9c-438f-829e-976160a5b1dc","Type":"ContainerStarted","Data":"2ba046929c6ce68d7446f34a0055244c965f655629778ad8c4dcf3396025a59b"} Jul 08 18:13:42.503135 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:42.502781 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" Jul 08 18:13:42.528772 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:42.528712 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" podStartSLOduration=2.078154596 podStartE2EDuration="2.52869176s" totalImagesPullingTime="450.537164ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:13:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:13:42.525207276 +0000 UTC m=+2098.209304606" watchObservedRunningTime="2026-07-08 18:13:42.52869176 +0000 UTC m=+2098.212789039" Jul 08 18:13:44.515390 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:44.515356 2584 generic.go:356] "Generic (PLEG): container finished" podID="f4635485-8278-4278-988b-4b889ddb42eb" containerID="c741da0cc2cdbd38702babe96719f8d6ac40e7591602419f64cd96ac9d83e21f" exitCode=0 Jul 08 18:13:44.515808 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:44.515430 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" event={"ID":"f4635485-8278-4278-988b-4b889ddb42eb","Type":"ContainerDied","Data":"c741da0cc2cdbd38702babe96719f8d6ac40e7591602419f64cd96ac9d83e21f"} Jul 08 18:13:44.515852 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:44.515817 2584 scope.go:117] "RemoveContainer" containerID="c741da0cc2cdbd38702babe96719f8d6ac40e7591602419f64cd96ac9d83e21f" Jul 08 18:13:44.516807 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:44.516780 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 18:13:44.516862 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:44.516846 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 18:13:44.516897 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:44.516879 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 18:13:44.602889 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:44.602841 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:45.017840 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:45.017808 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/1.log" Jul 08 18:13:45.023491 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:45.023465 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/1.log" Jul 08 18:13:45.426804 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:45.426771 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:45.522968 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:45.522925 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" event={"ID":"f4635485-8278-4278-988b-4b889ddb42eb","Type":"ContainerStarted","Data":"4204e543fa7a62333bbe5633627ec6ac2eca3ab5753f76fe884473d94ccfffa8"} Jul 08 18:13:46.283360 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:46.283316 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:46.284669 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:46.284642 2584 patch_prober.go:28] interesting pod/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 08 18:13:46.284800 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:46.284690 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" podUID="f4635485-8278-4278-988b-4b889ddb42eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 08 18:13:47.283476 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:47.283442 2584 patch_prober.go:28] interesting pod/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 08 18:13:47.283905 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:47.283496 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" podUID="f4635485-8278-4278-988b-4b889ddb42eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 08 18:13:48.283731 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:48.283693 2584 patch_prober.go:28] interesting pod/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 08 18:13:48.284307 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:48.283767 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" podUID="f4635485-8278-4278-988b-4b889ddb42eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 08 18:13:48.513452 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:48.513422 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" Jul 08 18:13:49.283658 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:49.283616 2584 patch_prober.go:28] interesting pod/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 08 18:13:49.283872 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:49.283678 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" podUID="f4635485-8278-4278-988b-4b889ddb42eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 08 18:13:50.287222 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:50.287192 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:50.287643 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:50.287444 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:50.288218 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:50.288201 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:13:52.889513 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:52.889468 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:53.658292 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:53.658257 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc"] Jul 08 18:13:53.658524 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:53.658499 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" podUID="ed9fefee-0d6e-4804-9ba5-7c720cd7601b" containerName="maas-api" containerID="cri-o://56f114c807d78cd2da39b51c1c95bdf85864b59658872f33f7c5785dc5546d4d" gracePeriod=30 Jul 08 18:13:53.909786 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:53.909733 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" Jul 08 18:13:53.925864 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:53.925840 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbv2c\" (UniqueName: \"kubernetes.io/projected/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-kube-api-access-gbv2c\") pod \"ed9fefee-0d6e-4804-9ba5-7c720cd7601b\" (UID: \"ed9fefee-0d6e-4804-9ba5-7c720cd7601b\") " Jul 08 18:13:53.926023 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:53.925871 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-maas-api-tls\") pod \"ed9fefee-0d6e-4804-9ba5-7c720cd7601b\" (UID: \"ed9fefee-0d6e-4804-9ba5-7c720cd7601b\") " Jul 08 18:13:53.926023 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:53.925905 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-openshift-service-ca\") pod \"ed9fefee-0d6e-4804-9ba5-7c720cd7601b\" (UID: \"ed9fefee-0d6e-4804-9ba5-7c720cd7601b\") " Jul 08 18:13:53.926264 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:53.926225 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ed9fefee-0d6e-4804-9ba5-7c720cd7601b" (UID: "ed9fefee-0d6e-4804-9ba5-7c720cd7601b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:13:53.928716 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:53.928673 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-kube-api-access-gbv2c" (OuterVolumeSpecName: "kube-api-access-gbv2c") pod "ed9fefee-0d6e-4804-9ba5-7c720cd7601b" (UID: "ed9fefee-0d6e-4804-9ba5-7c720cd7601b"). InnerVolumeSpecName "kube-api-access-gbv2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:13:53.935290 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:53.935234 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ed9fefee-0d6e-4804-9ba5-7c720cd7601b" (UID: "ed9fefee-0d6e-4804-9ba5-7c720cd7601b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:13:53.989564 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:53.989525 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:54.027270 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:54.026759 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-openshift-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:13:54.027270 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:54.026806 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbv2c\" (UniqueName: \"kubernetes.io/projected/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-kube-api-access-gbv2c\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:13:54.027270 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:54.026821 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed9fefee-0d6e-4804-9ba5-7c720cd7601b-maas-api-tls\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:13:54.090357 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:54.090319 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:54.584361 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:54.584324 2584 generic.go:356] "Generic (PLEG): container finished" podID="ed9fefee-0d6e-4804-9ba5-7c720cd7601b" containerID="56f114c807d78cd2da39b51c1c95bdf85864b59658872f33f7c5785dc5546d4d" exitCode=0 Jul 08 18:13:54.584539 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:54.584374 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" Jul 08 18:13:54.584539 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:54.584414 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" event={"ID":"ed9fefee-0d6e-4804-9ba5-7c720cd7601b","Type":"ContainerDied","Data":"56f114c807d78cd2da39b51c1c95bdf85864b59658872f33f7c5785dc5546d4d"} Jul 08 18:13:54.584539 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:54.584471 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc" event={"ID":"ed9fefee-0d6e-4804-9ba5-7c720cd7601b","Type":"ContainerDied","Data":"24b22a3d8e0f3e823f1b0e721f1d613d6ba61bd8c29fb8f7bb951e57c347ccfa"} Jul 08 18:13:54.584539 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:54.584502 2584 scope.go:117] "RemoveContainer" containerID="56f114c807d78cd2da39b51c1c95bdf85864b59658872f33f7c5785dc5546d4d" Jul 08 18:13:54.594648 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:54.594629 2584 scope.go:117] "RemoveContainer" containerID="56f114c807d78cd2da39b51c1c95bdf85864b59658872f33f7c5785dc5546d4d" Jul 08 18:13:54.594909 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:13:54.594888 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f114c807d78cd2da39b51c1c95bdf85864b59658872f33f7c5785dc5546d4d\": container with ID starting with 56f114c807d78cd2da39b51c1c95bdf85864b59658872f33f7c5785dc5546d4d not found: ID does not exist" containerID="56f114c807d78cd2da39b51c1c95bdf85864b59658872f33f7c5785dc5546d4d" Jul 08 18:13:54.594954 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:54.594919 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f114c807d78cd2da39b51c1c95bdf85864b59658872f33f7c5785dc5546d4d"} err="failed to get container status \"56f114c807d78cd2da39b51c1c95bdf85864b59658872f33f7c5785dc5546d4d\": rpc error: code = NotFound desc = could not find container \"56f114c807d78cd2da39b51c1c95bdf85864b59658872f33f7c5785dc5546d4d\": container with ID starting with 56f114c807d78cd2da39b51c1c95bdf85864b59658872f33f7c5785dc5546d4d not found: ID does not exist" Jul 08 18:13:54.610937 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:54.610906 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc"] Jul 08 18:13:54.615647 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:54.615621 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-988034-9c57ff54-b6wkc"] Jul 08 18:13:54.957296 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:54.957263 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed9fefee-0d6e-4804-9ba5-7c720cd7601b" path="/var/lib/kubelet/pods/ed9fefee-0d6e-4804-9ba5-7c720cd7601b/volumes" Jul 08 18:13:59.608426 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:59.608388 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:59.708650 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:59.708615 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:13:59.875281 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:59.875231 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4"] Jul 08 18:13:59.875474 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:59.875452 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" podUID="f4635485-8278-4278-988b-4b889ddb42eb" containerName="istio-proxy" containerID="cri-o://4204e543fa7a62333bbe5633627ec6ac2eca3ab5753f76fe884473d94ccfffa8" gracePeriod=30 Jul 08 18:13:59.889443 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:13:59.889408 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:00.411533 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:00.411499 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:00.864289 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:00.864255 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:00.993688 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:00.993645 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:01.094306 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.094273 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:01.278983 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.278898 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l"] Jul 08 18:14:01.279262 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.279219 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" podUID="6822cc38-4f9c-438f-829e-976160a5b1dc" containerName="maas-api" containerID="cri-o://2ba046929c6ce68d7446f34a0055244c965f655629778ad8c4dcf3396025a59b" gracePeriod=30 Jul 08 18:14:01.301762 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.301731 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:01.542635 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.542571 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" Jul 08 18:14:01.595852 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.595820 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6822cc38-4f9c-438f-829e-976160a5b1dc-openshift-service-ca\") pod \"6822cc38-4f9c-438f-829e-976160a5b1dc\" (UID: \"6822cc38-4f9c-438f-829e-976160a5b1dc\") " Jul 08 18:14:01.596050 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.595877 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdb5c\" (UniqueName: \"kubernetes.io/projected/6822cc38-4f9c-438f-829e-976160a5b1dc-kube-api-access-bdb5c\") pod \"6822cc38-4f9c-438f-829e-976160a5b1dc\" (UID: \"6822cc38-4f9c-438f-829e-976160a5b1dc\") " Jul 08 18:14:01.596050 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.595932 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6822cc38-4f9c-438f-829e-976160a5b1dc-maas-api-tls\") pod \"6822cc38-4f9c-438f-829e-976160a5b1dc\" (UID: \"6822cc38-4f9c-438f-829e-976160a5b1dc\") " Jul 08 18:14:01.596297 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.596271 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6822cc38-4f9c-438f-829e-976160a5b1dc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6822cc38-4f9c-438f-829e-976160a5b1dc" (UID: "6822cc38-4f9c-438f-829e-976160a5b1dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:14:01.598279 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.598233 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6822cc38-4f9c-438f-829e-976160a5b1dc-kube-api-access-bdb5c" (OuterVolumeSpecName: "kube-api-access-bdb5c") pod "6822cc38-4f9c-438f-829e-976160a5b1dc" (UID: "6822cc38-4f9c-438f-829e-976160a5b1dc"). InnerVolumeSpecName "kube-api-access-bdb5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:14:01.598592 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.598560 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6822cc38-4f9c-438f-829e-976160a5b1dc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6822cc38-4f9c-438f-829e-976160a5b1dc" (UID: "6822cc38-4f9c-438f-829e-976160a5b1dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:14:01.626271 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.626213 2584 generic.go:356] "Generic (PLEG): container finished" podID="6822cc38-4f9c-438f-829e-976160a5b1dc" containerID="2ba046929c6ce68d7446f34a0055244c965f655629778ad8c4dcf3396025a59b" exitCode=0 Jul 08 18:14:01.626446 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.626321 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" event={"ID":"6822cc38-4f9c-438f-829e-976160a5b1dc","Type":"ContainerDied","Data":"2ba046929c6ce68d7446f34a0055244c965f655629778ad8c4dcf3396025a59b"} Jul 08 18:14:01.626446 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.626362 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" event={"ID":"6822cc38-4f9c-438f-829e-976160a5b1dc","Type":"ContainerDied","Data":"837139f00471f2b9069e37b77a6e14d1743becdb4dcec32e12d1bc8a09b51eb3"} Jul 08 18:14:01.626446 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.626388 2584 scope.go:117] "RemoveContainer" containerID="2ba046929c6ce68d7446f34a0055244c965f655629778ad8c4dcf3396025a59b" Jul 08 18:14:01.626446 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.626321 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l" Jul 08 18:14:01.637274 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.637229 2584 scope.go:117] "RemoveContainer" containerID="2ba046929c6ce68d7446f34a0055244c965f655629778ad8c4dcf3396025a59b" Jul 08 18:14:01.637566 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:14:01.637543 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba046929c6ce68d7446f34a0055244c965f655629778ad8c4dcf3396025a59b\": container with ID starting with 2ba046929c6ce68d7446f34a0055244c965f655629778ad8c4dcf3396025a59b not found: ID does not exist" containerID="2ba046929c6ce68d7446f34a0055244c965f655629778ad8c4dcf3396025a59b" Jul 08 18:14:01.637676 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.637578 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba046929c6ce68d7446f34a0055244c965f655629778ad8c4dcf3396025a59b"} err="failed to get container status \"2ba046929c6ce68d7446f34a0055244c965f655629778ad8c4dcf3396025a59b\": rpc error: code = NotFound desc = could not find container \"2ba046929c6ce68d7446f34a0055244c965f655629778ad8c4dcf3396025a59b\": container with ID starting with 2ba046929c6ce68d7446f34a0055244c965f655629778ad8c4dcf3396025a59b not found: ID does not exist" Jul 08 18:14:01.652484 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.652449 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l"] Jul 08 18:14:01.656823 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.656799 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1dc085-8585b7fd74-2z75l"] Jul 08 18:14:01.696974 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.696937 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6822cc38-4f9c-438f-829e-976160a5b1dc-openshift-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:14:01.696974 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.696967 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdb5c\" (UniqueName: \"kubernetes.io/projected/6822cc38-4f9c-438f-829e-976160a5b1dc-kube-api-access-bdb5c\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:14:01.697169 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:01.696988 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6822cc38-4f9c-438f-829e-976160a5b1dc-maas-api-tls\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:14:02.956846 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:02.956809 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6822cc38-4f9c-438f-829e-976160a5b1dc" path="/var/lib/kubelet/pods/6822cc38-4f9c-438f-829e-976160a5b1dc/volumes" Jul 08 18:14:05.146710 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.146639 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:14:05.228227 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.228193 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4635485-8278-4278-988b-4b889ddb42eb-istio-token\") pod \"f4635485-8278-4278-988b-4b889ddb42eb\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " Jul 08 18:14:05.228227 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.228229 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4635485-8278-4278-988b-4b889ddb42eb-istiod-ca-cert\") pod \"f4635485-8278-4278-988b-4b889ddb42eb\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " Jul 08 18:14:05.228518 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.228293 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-istio-data\") pod \"f4635485-8278-4278-988b-4b889ddb42eb\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " Jul 08 18:14:05.228518 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.228314 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4635485-8278-4278-988b-4b889ddb42eb-istio-podinfo\") pod \"f4635485-8278-4278-988b-4b889ddb42eb\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " Jul 08 18:14:05.228518 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.228345 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-credential-socket\") pod \"f4635485-8278-4278-988b-4b889ddb42eb\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " Jul 08 18:14:05.228518 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.228376 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhr84\" (UniqueName: \"kubernetes.io/projected/f4635485-8278-4278-988b-4b889ddb42eb-kube-api-access-vhr84\") pod \"f4635485-8278-4278-988b-4b889ddb42eb\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " Jul 08 18:14:05.228518 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.228399 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-workload-certs\") pod \"f4635485-8278-4278-988b-4b889ddb42eb\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " Jul 08 18:14:05.228518 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.228434 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-workload-socket\") pod \"f4635485-8278-4278-988b-4b889ddb42eb\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " Jul 08 18:14:05.228518 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.228484 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-istio-envoy\") pod \"f4635485-8278-4278-988b-4b889ddb42eb\" (UID: \"f4635485-8278-4278-988b-4b889ddb42eb\") " Jul 08 18:14:05.228874 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.228678 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f4635485-8278-4278-988b-4b889ddb42eb" (UID: "f4635485-8278-4278-988b-4b889ddb42eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:14:05.228874 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.228812 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-credential-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:14:05.229131 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.228995 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f4635485-8278-4278-988b-4b889ddb42eb" (UID: "f4635485-8278-4278-988b-4b889ddb42eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:14:05.229227 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.229159 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f4635485-8278-4278-988b-4b889ddb42eb" (UID: "f4635485-8278-4278-988b-4b889ddb42eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:14:05.229462 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.229417 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-istio-data" (OuterVolumeSpecName: "istio-data") pod "f4635485-8278-4278-988b-4b889ddb42eb" (UID: "f4635485-8278-4278-988b-4b889ddb42eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:14:05.229620 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.229594 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4635485-8278-4278-988b-4b889ddb42eb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f4635485-8278-4278-988b-4b889ddb42eb" (UID: "f4635485-8278-4278-988b-4b889ddb42eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:14:05.230739 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.230718 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f4635485-8278-4278-988b-4b889ddb42eb" (UID: "f4635485-8278-4278-988b-4b889ddb42eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:14:05.230825 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.230743 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4635485-8278-4278-988b-4b889ddb42eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "f4635485-8278-4278-988b-4b889ddb42eb" (UID: "f4635485-8278-4278-988b-4b889ddb42eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:14:05.230971 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.230948 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4635485-8278-4278-988b-4b889ddb42eb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f4635485-8278-4278-988b-4b889ddb42eb" (UID: "f4635485-8278-4278-988b-4b889ddb42eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 18:14:05.231612 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.231589 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4635485-8278-4278-988b-4b889ddb42eb-kube-api-access-vhr84" (OuterVolumeSpecName: "kube-api-access-vhr84") pod "f4635485-8278-4278-988b-4b889ddb42eb" (UID: "f4635485-8278-4278-988b-4b889ddb42eb"). InnerVolumeSpecName "kube-api-access-vhr84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:14:05.329580 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.329531 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4635485-8278-4278-988b-4b889ddb42eb-istio-token\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:14:05.329580 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.329581 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4635485-8278-4278-988b-4b889ddb42eb-istiod-ca-cert\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:14:05.329580 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.329594 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-istio-data\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:14:05.329829 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.329605 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4635485-8278-4278-988b-4b889ddb42eb-istio-podinfo\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:14:05.329829 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.329617 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhr84\" (UniqueName: \"kubernetes.io/projected/f4635485-8278-4278-988b-4b889ddb42eb-kube-api-access-vhr84\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:14:05.329829 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.329630 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-workload-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:14:05.329829 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.329642 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-workload-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:14:05.329829 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.329655 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4635485-8278-4278-988b-4b889ddb42eb-istio-envoy\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:14:05.653221 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.653187 2584 generic.go:356] "Generic (PLEG): container finished" podID="f4635485-8278-4278-988b-4b889ddb42eb" containerID="4204e543fa7a62333bbe5633627ec6ac2eca3ab5753f76fe884473d94ccfffa8" exitCode=0 Jul 08 18:14:05.653416 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.653261 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" Jul 08 18:14:05.653416 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.653287 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" event={"ID":"f4635485-8278-4278-988b-4b889ddb42eb","Type":"ContainerDied","Data":"4204e543fa7a62333bbe5633627ec6ac2eca3ab5753f76fe884473d94ccfffa8"} Jul 08 18:14:05.653416 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.653352 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4" event={"ID":"f4635485-8278-4278-988b-4b889ddb42eb","Type":"ContainerDied","Data":"fdd2e30869c3397cff9ab54a6e188f806ca637ee03f43ada32cabd1c6dc44b22"} Jul 08 18:14:05.653416 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.653382 2584 scope.go:117] "RemoveContainer" containerID="4204e543fa7a62333bbe5633627ec6ac2eca3ab5753f76fe884473d94ccfffa8" Jul 08 18:14:05.663149 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.663131 2584 scope.go:117] "RemoveContainer" containerID="c741da0cc2cdbd38702babe96719f8d6ac40e7591602419f64cd96ac9d83e21f" Jul 08 18:14:05.672123 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.672105 2584 scope.go:117] "RemoveContainer" containerID="4204e543fa7a62333bbe5633627ec6ac2eca3ab5753f76fe884473d94ccfffa8" Jul 08 18:14:05.672513 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:14:05.672490 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4204e543fa7a62333bbe5633627ec6ac2eca3ab5753f76fe884473d94ccfffa8\": container with ID starting with 4204e543fa7a62333bbe5633627ec6ac2eca3ab5753f76fe884473d94ccfffa8 not found: ID does not exist" containerID="4204e543fa7a62333bbe5633627ec6ac2eca3ab5753f76fe884473d94ccfffa8" Jul 08 18:14:05.672611 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.672522 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4204e543fa7a62333bbe5633627ec6ac2eca3ab5753f76fe884473d94ccfffa8"} err="failed to get container status \"4204e543fa7a62333bbe5633627ec6ac2eca3ab5753f76fe884473d94ccfffa8\": rpc error: code = NotFound desc = could not find container \"4204e543fa7a62333bbe5633627ec6ac2eca3ab5753f76fe884473d94ccfffa8\": container with ID starting with 4204e543fa7a62333bbe5633627ec6ac2eca3ab5753f76fe884473d94ccfffa8 not found: ID does not exist" Jul 08 18:14:05.672611 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.672540 2584 scope.go:117] "RemoveContainer" containerID="c741da0cc2cdbd38702babe96719f8d6ac40e7591602419f64cd96ac9d83e21f" Jul 08 18:14:05.672796 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:14:05.672779 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c741da0cc2cdbd38702babe96719f8d6ac40e7591602419f64cd96ac9d83e21f\": container with ID starting with c741da0cc2cdbd38702babe96719f8d6ac40e7591602419f64cd96ac9d83e21f not found: ID does not exist" containerID="c741da0cc2cdbd38702babe96719f8d6ac40e7591602419f64cd96ac9d83e21f" Jul 08 18:14:05.672838 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.672804 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c741da0cc2cdbd38702babe96719f8d6ac40e7591602419f64cd96ac9d83e21f"} err="failed to get container status \"c741da0cc2cdbd38702babe96719f8d6ac40e7591602419f64cd96ac9d83e21f\": rpc error: code = NotFound desc = could not find container \"c741da0cc2cdbd38702babe96719f8d6ac40e7591602419f64cd96ac9d83e21f\": container with ID starting with c741da0cc2cdbd38702babe96719f8d6ac40e7591602419f64cd96ac9d83e21f not found: ID does not exist" Jul 08 18:14:05.677189 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.677165 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4"] Jul 08 18:14:05.681997 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:05.681972 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-988034-openshift-default-5949bbb7-h8lc4"] Jul 08 18:14:06.956948 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:06.956913 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4635485-8278-4278-988b-4b889ddb42eb" path="/var/lib/kubelet/pods/f4635485-8278-4278-988b-4b889ddb42eb/volumes" Jul 08 18:14:07.179624 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:07.179585 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:07.283156 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:07.283066 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:07.486072 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:07.486032 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:08.023712 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:08.023680 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:08.366210 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:08.366164 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:08.674075 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:08.673985 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:11.204379 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.204333 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:11.281644 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.281580 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:11.282984 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.282960 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz"] Jul 08 18:14:11.283567 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.283546 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4635485-8278-4278-988b-4b889ddb42eb" containerName="istio-proxy" Jul 08 18:14:11.283631 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.283573 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4635485-8278-4278-988b-4b889ddb42eb" containerName="istio-proxy" Jul 08 18:14:11.283631 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.283618 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed9fefee-0d6e-4804-9ba5-7c720cd7601b" containerName="maas-api" Jul 08 18:14:11.283631 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.283626 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed9fefee-0d6e-4804-9ba5-7c720cd7601b" containerName="maas-api" Jul 08 18:14:11.283730 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.283644 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6822cc38-4f9c-438f-829e-976160a5b1dc" containerName="maas-api" Jul 08 18:14:11.283730 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.283652 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="6822cc38-4f9c-438f-829e-976160a5b1dc" containerName="maas-api" Jul 08 18:14:11.283790 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.283749 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed9fefee-0d6e-4804-9ba5-7c720cd7601b" containerName="maas-api" Jul 08 18:14:11.283790 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.283763 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="6822cc38-4f9c-438f-829e-976160a5b1dc" containerName="maas-api" Jul 08 18:14:11.283790 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.283772 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4635485-8278-4278-988b-4b889ddb42eb" containerName="istio-proxy" Jul 08 18:14:11.283790 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.283783 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4635485-8278-4278-988b-4b889ddb42eb" containerName="istio-proxy" Jul 08 18:14:11.288560 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.288543 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" Jul 08 18:14:11.291511 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.291494 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-1c14cf-serving-cert\"" Jul 08 18:14:11.303453 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.303433 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz"] Jul 08 18:14:11.357059 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.357030 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:11.381173 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.381139 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fac69956-a30b-4da8-b091-1fcc487d71ac-maas-api-tls\") pod \"maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz\" (UID: \"fac69956-a30b-4da8-b091-1fcc487d71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" Jul 08 18:14:11.381325 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.381182 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bb2x\" (UniqueName: \"kubernetes.io/projected/fac69956-a30b-4da8-b091-1fcc487d71ac-kube-api-access-5bb2x\") pod \"maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz\" (UID: \"fac69956-a30b-4da8-b091-1fcc487d71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" Jul 08 18:14:11.381325 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.381205 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fac69956-a30b-4da8-b091-1fcc487d71ac-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz\" (UID: \"fac69956-a30b-4da8-b091-1fcc487d71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" Jul 08 18:14:11.414776 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.414744 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:11.482380 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.482291 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fac69956-a30b-4da8-b091-1fcc487d71ac-maas-api-tls\") pod \"maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz\" (UID: \"fac69956-a30b-4da8-b091-1fcc487d71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" Jul 08 18:14:11.482380 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.482341 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bb2x\" (UniqueName: \"kubernetes.io/projected/fac69956-a30b-4da8-b091-1fcc487d71ac-kube-api-access-5bb2x\") pod \"maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz\" (UID: \"fac69956-a30b-4da8-b091-1fcc487d71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" Jul 08 18:14:11.482380 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.482374 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fac69956-a30b-4da8-b091-1fcc487d71ac-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz\" (UID: \"fac69956-a30b-4da8-b091-1fcc487d71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" Jul 08 18:14:11.483078 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.483054 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fac69956-a30b-4da8-b091-1fcc487d71ac-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz\" (UID: \"fac69956-a30b-4da8-b091-1fcc487d71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" Jul 08 18:14:11.484798 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.484778 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fac69956-a30b-4da8-b091-1fcc487d71ac-maas-api-tls\") pod \"maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz\" (UID: \"fac69956-a30b-4da8-b091-1fcc487d71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" Jul 08 18:14:11.499417 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.499393 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bb2x\" (UniqueName: \"kubernetes.io/projected/fac69956-a30b-4da8-b091-1fcc487d71ac-kube-api-access-5bb2x\") pod \"maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz\" (UID: \"fac69956-a30b-4da8-b091-1fcc487d71ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" Jul 08 18:14:11.598384 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.598352 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" Jul 08 18:14:11.747803 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.747770 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz"] Jul 08 18:14:11.749949 ip-10-0-129-226 kubenswrapper[2584]: W0708 18:14:11.749919 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac69956_a30b_4da8_b091_1fcc487d71ac.slice/crio-99997602d48c7f34b328fab75fbbd1036aa0483ac6639076ec4f0f895bbc3f62 WatchSource:0}: Error finding container 99997602d48c7f34b328fab75fbbd1036aa0483ac6639076ec4f0f895bbc3f62: Status 404 returned error can't find the container with id 99997602d48c7f34b328fab75fbbd1036aa0483ac6639076ec4f0f895bbc3f62 Jul 08 18:14:11.912503 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:11.912455 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:12.012108 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:12.012015 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:12.694555 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:12.694520 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" event={"ID":"fac69956-a30b-4da8-b091-1fcc487d71ac","Type":"ContainerStarted","Data":"8973316fe7a595dea3e7dc1b85929dfdd525a59485c32ee78b00ffb814f699ca"} Jul 08 18:14:12.694986 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:12.694575 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" event={"ID":"fac69956-a30b-4da8-b091-1fcc487d71ac","Type":"ContainerStarted","Data":"99997602d48c7f34b328fab75fbbd1036aa0483ac6639076ec4f0f895bbc3f62"} Jul 08 18:14:12.694986 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:12.694597 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" Jul 08 18:14:12.714609 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:12.714561 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" podStartSLOduration=0.931784912 podStartE2EDuration="1.714544952s" totalImagesPullingTime="782.76004ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:14:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:14:12.713388547 +0000 UTC m=+2128.397485827" watchObservedRunningTime="2026-07-08 18:14:12.714544952 +0000 UTC m=+2128.398642228" Jul 08 18:14:15.692390 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:15.692340 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:16.044864 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.044779 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:16.336034 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.335959 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj"] Jul 08 18:14:16.336369 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.336356 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4635485-8278-4278-988b-4b889ddb42eb" containerName="istio-proxy" Jul 08 18:14:16.336412 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.336370 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4635485-8278-4278-988b-4b889ddb42eb" containerName="istio-proxy" Jul 08 18:14:16.339668 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.339644 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.342320 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.342300 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-d1aec6-openshift-default-dockercfg-tcrvb\"" Jul 08 18:14:16.342421 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.342309 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 18:14:16.358209 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.358183 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj"] Jul 08 18:14:16.364352 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.364309 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:16.427677 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.427642 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-workload-socket\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.427880 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.427691 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-workload-certs\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.427880 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.427789 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-envoy\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.427880 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.427829 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nffj5\" (UniqueName: \"kubernetes.io/projected/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-kube-api-access-nffj5\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.428050 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.427883 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-data\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.428050 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.427919 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-credential-socket\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.428050 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.427949 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-token\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.428050 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.427986 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-podinfo\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.428050 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.428041 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istiod-ca-cert\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.528803 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.528767 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-workload-socket\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.528803 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.528811 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-workload-certs\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.529054 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.528926 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-envoy\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.529116 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.529047 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nffj5\" (UniqueName: \"kubernetes.io/projected/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-kube-api-access-nffj5\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.529173 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.529136 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-data\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.529228 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.529175 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-credential-socket\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.529323 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.529225 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-token\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.529323 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.529286 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-workload-socket\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.529323 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.529305 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-podinfo\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.529479 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.529387 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istiod-ca-cert\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.529479 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.529467 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-data\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.529577 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.529225 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-workload-certs\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.529765 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.529740 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-credential-socket\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.530133 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.530113 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istiod-ca-cert\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.532179 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.532155 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-envoy\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.532404 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.532385 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-podinfo\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.540022 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.539990 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nffj5\" (UniqueName: \"kubernetes.io/projected/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-kube-api-access-nffj5\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.540398 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.540368 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-token\") pod \"e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.650550 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.650519 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:16.718783 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.718744 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:16.805008 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.804981 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj"] Jul 08 18:14:16.806177 ip-10-0-129-226 kubenswrapper[2584]: W0708 18:14:16.806153 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5f4bc8_d83f_4f17_91f3_3cdce596fb0a.slice/crio-8bcaac66b13787404a666b489aead9e72ca4c88f2de48171f3723037e75af408 WatchSource:0}: Error finding container 8bcaac66b13787404a666b489aead9e72ca4c88f2de48171f3723037e75af408: Status 404 returned error can't find the container with id 8bcaac66b13787404a666b489aead9e72ca4c88f2de48171f3723037e75af408 Jul 08 18:14:16.808475 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.808441 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 18:14:16.808590 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.808521 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 18:14:16.808590 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:16.808566 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 18:14:17.731665 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:17.731625 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" event={"ID":"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a","Type":"ContainerStarted","Data":"31fe619dba40ae7cbef6510b40897ae019e96b5e39e4e2bb77615790cb681a85"} Jul 08 18:14:17.732154 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:17.731688 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" event={"ID":"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a","Type":"ContainerStarted","Data":"8bcaac66b13787404a666b489aead9e72ca4c88f2de48171f3723037e75af408"} Jul 08 18:14:17.760905 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:17.760853 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" podStartSLOduration=1.7608394139999999 podStartE2EDuration="1.760839414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:14:17.757615857 +0000 UTC m=+2133.441713134" watchObservedRunningTime="2026-07-08 18:14:17.760839414 +0000 UTC m=+2133.444936748" Jul 08 18:14:18.651771 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:18.651732 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:18.656839 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:18.656812 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:18.709137 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:18.709112 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" Jul 08 18:14:18.737681 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:18.737639 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:18.738471 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:18.738450 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:19.680756 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:19.680718 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:19.762809 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:19.762759 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:19.764551 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:19.764521 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw"] Jul 08 18:14:19.768447 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:19.768428 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" Jul 08 18:14:19.770998 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:19.770977 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-d1aec6-serving-cert\"" Jul 08 18:14:19.781062 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:19.781037 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw"] Jul 08 18:14:19.839667 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:19.839633 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:19.857160 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:19.857132 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8tdr\" (UniqueName: \"kubernetes.io/projected/ff591338-f323-457a-95ae-41f146f5fdd2-kube-api-access-k8tdr\") pod \"maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw\" (UID: \"ff591338-f323-457a-95ae-41f146f5fdd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" Jul 08 18:14:19.857160 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:19.857168 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff591338-f323-457a-95ae-41f146f5fdd2-maas-api-tls\") pod \"maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw\" (UID: \"ff591338-f323-457a-95ae-41f146f5fdd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" Jul 08 18:14:19.857424 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:19.857324 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff591338-f323-457a-95ae-41f146f5fdd2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw\" (UID: \"ff591338-f323-457a-95ae-41f146f5fdd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" Jul 08 18:14:19.958636 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:19.958554 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff591338-f323-457a-95ae-41f146f5fdd2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw\" (UID: \"ff591338-f323-457a-95ae-41f146f5fdd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" Jul 08 18:14:19.958636 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:19.958619 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8tdr\" (UniqueName: \"kubernetes.io/projected/ff591338-f323-457a-95ae-41f146f5fdd2-kube-api-access-k8tdr\") pod \"maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw\" (UID: \"ff591338-f323-457a-95ae-41f146f5fdd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" Jul 08 18:14:19.958820 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:19.958648 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff591338-f323-457a-95ae-41f146f5fdd2-maas-api-tls\") pod \"maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw\" (UID: \"ff591338-f323-457a-95ae-41f146f5fdd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" Jul 08 18:14:19.958820 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:14:19.958780 2584 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-serving-cert: secret "maas-api-e2e-shared-b-d1aec6-serving-cert" not found Jul 08 18:14:19.958891 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:14:19.958854 2584 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff591338-f323-457a-95ae-41f146f5fdd2-maas-api-tls podName:ff591338-f323-457a-95ae-41f146f5fdd2 nodeName:}" failed. No retries permitted until 2026-07-08 18:14:20.458835719 +0000 UTC m=+2136.142932973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff591338-f323-457a-95ae-41f146f5fdd2-maas-api-tls") pod "maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" (UID: "ff591338-f323-457a-95ae-41f146f5fdd2") : secret "maas-api-e2e-shared-b-d1aec6-serving-cert" not found Jul 08 18:14:19.959226 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:19.959206 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff591338-f323-457a-95ae-41f146f5fdd2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw\" (UID: \"ff591338-f323-457a-95ae-41f146f5fdd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" Jul 08 18:14:19.968343 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:19.968312 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8tdr\" (UniqueName: \"kubernetes.io/projected/ff591338-f323-457a-95ae-41f146f5fdd2-kube-api-access-k8tdr\") pod \"maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw\" (UID: \"ff591338-f323-457a-95ae-41f146f5fdd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" Jul 08 18:14:20.149746 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:20.149713 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:20.290302 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:20.290197 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:20.391546 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:20.391507 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:20.463939 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:20.463895 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff591338-f323-457a-95ae-41f146f5fdd2-maas-api-tls\") pod \"maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw\" (UID: \"ff591338-f323-457a-95ae-41f146f5fdd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" Jul 08 18:14:20.467035 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:20.466990 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff591338-f323-457a-95ae-41f146f5fdd2-maas-api-tls\") pod \"maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw\" (UID: \"ff591338-f323-457a-95ae-41f146f5fdd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" Jul 08 18:14:20.679845 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:20.679808 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" Jul 08 18:14:20.827217 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:20.827188 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw"] Jul 08 18:14:20.831292 ip-10-0-129-226 kubenswrapper[2584]: W0708 18:14:20.828092 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff591338_f323_457a_95ae_41f146f5fdd2.slice/crio-3291b82f2aa149bc3d301dbe885f8ba855d37fc052d0cf319a6cae65d91f7986 WatchSource:0}: Error finding container 3291b82f2aa149bc3d301dbe885f8ba855d37fc052d0cf319a6cae65d91f7986: Status 404 returned error can't find the container with id 3291b82f2aa149bc3d301dbe885f8ba855d37fc052d0cf319a6cae65d91f7986 Jul 08 18:14:21.765389 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:21.765271 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" event={"ID":"ff591338-f323-457a-95ae-41f146f5fdd2","Type":"ContainerStarted","Data":"cfe638c8e0910d3548f8448fefdad25c9f70ad46732149895b67a0f2084bc2b7"} Jul 08 18:14:21.765389 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:21.765335 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" event={"ID":"ff591338-f323-457a-95ae-41f146f5fdd2","Type":"ContainerStarted","Data":"3291b82f2aa149bc3d301dbe885f8ba855d37fc052d0cf319a6cae65d91f7986"} Jul 08 18:14:21.765389 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:21.765368 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" Jul 08 18:14:21.785206 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:21.785152 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" podStartSLOduration=2.23879335 podStartE2EDuration="2.78513614s" totalImagesPullingTime="546.34279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:14:21.782737166 +0000 UTC m=+2137.466834455" watchObservedRunningTime="2026-07-08 18:14:21.78513614 +0000 UTC m=+2137.469233417" Jul 08 18:14:24.189326 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:24.189287 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:27.778488 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:27.778460 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" Jul 08 18:14:33.738732 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:33.738705 2584 patch_prober.go:28] interesting pod/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 08 18:14:33.739146 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:33.738764 2584 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" podUID="ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 08 18:14:33.837179 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:33.837145 2584 generic.go:356] "Generic (PLEG): container finished" podID="ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" containerID="31fe619dba40ae7cbef6510b40897ae019e96b5e39e4e2bb77615790cb681a85" exitCode=0 Jul 08 18:14:33.837387 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:33.837222 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" event={"ID":"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a","Type":"ContainerDied","Data":"31fe619dba40ae7cbef6510b40897ae019e96b5e39e4e2bb77615790cb681a85"} Jul 08 18:14:33.837658 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:33.837642 2584 scope.go:117] "RemoveContainer" containerID="31fe619dba40ae7cbef6510b40897ae019e96b5e39e4e2bb77615790cb681a85" Jul 08 18:14:33.838834 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:33.838797 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 18:14:33.838921 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:33.838889 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 18:14:33.838971 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:33.838933 2584 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 18:14:33.878770 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:33.878620 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:33.977402 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:33.977363 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:34.850293 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:34.850230 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" event={"ID":"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a","Type":"ContainerStarted","Data":"3432beecc819c12655850bc240396bb932e14be628510c4856aef93fef9b64ca"} Jul 08 18:14:35.650839 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:35.650794 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:35.651262 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:35.651208 2584 patch_prober.go:28] interesting pod/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 08 18:14:35.651476 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:35.651447 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" podUID="ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 08 18:14:36.651473 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:36.651434 2584 patch_prober.go:28] interesting pod/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 08 18:14:36.652087 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:36.652054 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" podUID="ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 08 18:14:37.651842 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:37.651330 2584 patch_prober.go:28] interesting pod/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 08 18:14:37.651842 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:37.651412 2584 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" podUID="ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 08 18:14:38.654740 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:38.654711 2584 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:38.655122 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:38.655042 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:38.655778 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:38.655757 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:14:42.477757 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:42.477709 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:14:42.577487 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:14:42.577449 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:15:00.146043 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.145998 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7"] Jul 08 18:15:00.150042 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.150004 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q"] Jul 08 18:15:00.150362 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.150339 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7" Jul 08 18:15:00.153806 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.153784 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725575-vzdhd"] Jul 08 18:15:00.153933 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.153913 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q" Jul 08 18:15:00.157260 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.157221 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-vzdhd" Jul 08 18:15:00.157959 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.157939 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7"] Jul 08 18:15:00.161436 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.161415 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q"] Jul 08 18:15:00.170967 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.170074 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725575-vzdhd"] Jul 08 18:15:00.237158 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.237118 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbcdm\" (UniqueName: \"kubernetes.io/projected/40999c7e-ada3-47b3-a6a0-cdd2c86d16c8-kube-api-access-mbcdm\") pod \"maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7\" (UID: \"40999c7e-ada3-47b3-a6a0-cdd2c86d16c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7" Jul 08 18:15:00.237353 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.237173 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c92j9\" (UniqueName: \"kubernetes.io/projected/c4ffdc8e-61db-44b2-a8fa-f9472a262523-kube-api-access-c92j9\") pod \"maas-api-key-cleanup-29725575-vzdhd\" (UID: \"c4ffdc8e-61db-44b2-a8fa-f9472a262523\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-vzdhd" Jul 08 18:15:00.237353 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.237299 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vbrd\" (UniqueName: \"kubernetes.io/projected/659af87c-0f4c-4e8f-a977-c02e06e128af-kube-api-access-4vbrd\") pod \"maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q\" (UID: \"659af87c-0f4c-4e8f-a977-c02e06e128af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q" Jul 08 18:15:00.338824 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.338784 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c92j9\" (UniqueName: \"kubernetes.io/projected/c4ffdc8e-61db-44b2-a8fa-f9472a262523-kube-api-access-c92j9\") pod \"maas-api-key-cleanup-29725575-vzdhd\" (UID: \"c4ffdc8e-61db-44b2-a8fa-f9472a262523\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-vzdhd" Jul 08 18:15:00.339016 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.338856 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vbrd\" (UniqueName: \"kubernetes.io/projected/659af87c-0f4c-4e8f-a977-c02e06e128af-kube-api-access-4vbrd\") pod \"maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q\" (UID: \"659af87c-0f4c-4e8f-a977-c02e06e128af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q" Jul 08 18:15:00.339016 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.338907 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbcdm\" (UniqueName: \"kubernetes.io/projected/40999c7e-ada3-47b3-a6a0-cdd2c86d16c8-kube-api-access-mbcdm\") pod \"maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7\" (UID: \"40999c7e-ada3-47b3-a6a0-cdd2c86d16c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7" Jul 08 18:15:00.363067 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.363031 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c92j9\" (UniqueName: \"kubernetes.io/projected/c4ffdc8e-61db-44b2-a8fa-f9472a262523-kube-api-access-c92j9\") pod \"maas-api-key-cleanup-29725575-vzdhd\" (UID: \"c4ffdc8e-61db-44b2-a8fa-f9472a262523\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-vzdhd" Jul 08 18:15:00.363263 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.363031 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vbrd\" (UniqueName: \"kubernetes.io/projected/659af87c-0f4c-4e8f-a977-c02e06e128af-kube-api-access-4vbrd\") pod \"maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q\" (UID: \"659af87c-0f4c-4e8f-a977-c02e06e128af\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q" Jul 08 18:15:00.364540 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.364513 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbcdm\" (UniqueName: \"kubernetes.io/projected/40999c7e-ada3-47b3-a6a0-cdd2c86d16c8-kube-api-access-mbcdm\") pod \"maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7\" (UID: \"40999c7e-ada3-47b3-a6a0-cdd2c86d16c8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7" Jul 08 18:15:00.468950 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.468854 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7" Jul 08 18:15:00.478161 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.478130 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q" Jul 08 18:15:00.496716 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.496493 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-vzdhd" Jul 08 18:15:00.644576 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.643873 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7"] Jul 08 18:15:00.699582 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.699558 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725575-vzdhd"] Jul 08 18:15:00.700189 ip-10-0-129-226 kubenswrapper[2584]: W0708 18:15:00.700166 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ffdc8e_61db_44b2_a8fa_f9472a262523.slice/crio-44443a4b7fa5375531351e2b076c3a584f4edf6d45b7ba55e5182375140e89f3 WatchSource:0}: Error finding container 44443a4b7fa5375531351e2b076c3a584f4edf6d45b7ba55e5182375140e89f3: Status 404 returned error can't find the container with id 44443a4b7fa5375531351e2b076c3a584f4edf6d45b7ba55e5182375140e89f3 Jul 08 18:15:00.881739 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:00.881675 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q"] Jul 08 18:15:00.884111 ip-10-0-129-226 kubenswrapper[2584]: W0708 18:15:00.884082 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659af87c_0f4c_4e8f_a977_c02e06e128af.slice/crio-16e4ded29f5fcf6c4b0d41c69980bb386cd269d6e65c70d190b180803e85e80c WatchSource:0}: Error finding container 16e4ded29f5fcf6c4b0d41c69980bb386cd269d6e65c70d190b180803e85e80c: Status 404 returned error can't find the container with id 16e4ded29f5fcf6c4b0d41c69980bb386cd269d6e65c70d190b180803e85e80c Jul 08 18:15:01.002303 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:01.002198 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q" event={"ID":"659af87c-0f4c-4e8f-a977-c02e06e128af","Type":"ContainerStarted","Data":"16e4ded29f5fcf6c4b0d41c69980bb386cd269d6e65c70d190b180803e85e80c"} Jul 08 18:15:01.003770 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:01.003740 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7" event={"ID":"40999c7e-ada3-47b3-a6a0-cdd2c86d16c8","Type":"ContainerStarted","Data":"b900843ca779c5f5730413eb4f705892a4dd9d5aeda3aa201ae5a3ece3148870"} Jul 08 18:15:01.003915 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:01.003794 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7" event={"ID":"40999c7e-ada3-47b3-a6a0-cdd2c86d16c8","Type":"ContainerStarted","Data":"d2410b6906af9e3e6411569e65b8b9ea77fe7d29fe739f5b3082dc195d722a54"} Jul 08 18:15:01.004955 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:01.004932 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-vzdhd" event={"ID":"c4ffdc8e-61db-44b2-a8fa-f9472a262523","Type":"ContainerStarted","Data":"44443a4b7fa5375531351e2b076c3a584f4edf6d45b7ba55e5182375140e89f3"} Jul 08 18:15:01.024792 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:01.024750 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7" podStartSLOduration=1.02473496 podStartE2EDuration="1.02473496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:15:01.023761306 +0000 UTC m=+2176.707858585" watchObservedRunningTime="2026-07-08 18:15:01.02473496 +0000 UTC m=+2176.708832239" Jul 08 18:15:02.012839 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:02.012801 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-vzdhd" event={"ID":"c4ffdc8e-61db-44b2-a8fa-f9472a262523","Type":"ContainerStarted","Data":"47cab145af4ebb34e719a5b7d5f15500c3c643459938f410cf1a5bdaaca39d01"} Jul 08 18:15:02.014520 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:02.014493 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q" event={"ID":"659af87c-0f4c-4e8f-a977-c02e06e128af","Type":"ContainerStarted","Data":"f1ba71317e87520a6ec564340b8045dda20e5abd1ac4f09af0bf3d73d9131429"} Jul 08 18:15:02.035923 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:02.035879 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-vzdhd" podStartSLOduration=2.035864333 podStartE2EDuration="2.035864333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:15:02.034141289 +0000 UTC m=+2177.718238566" watchObservedRunningTime="2026-07-08 18:15:02.035864333 +0000 UTC m=+2177.719961610" Jul 08 18:15:02.059362 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:02.059315 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q" podStartSLOduration=2.05929704 podStartE2EDuration="2.05929704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:15:02.05830713 +0000 UTC m=+2177.742404409" watchObservedRunningTime="2026-07-08 18:15:02.05929704 +0000 UTC m=+2177.743394317" Jul 08 18:15:57.945759 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:15:57.945719 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:16:00.017818 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:00.017768 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7"] Jul 08 18:16:00.018337 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:00.018082 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7" podUID="40999c7e-ada3-47b3-a6a0-cdd2c86d16c8" containerName="cleanup" containerID="cri-o://b900843ca779c5f5730413eb4f705892a4dd9d5aeda3aa201ae5a3ece3148870" gracePeriod=30 Jul 08 18:16:00.021262 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:00.021219 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q"] Jul 08 18:16:00.021514 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:00.021476 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q" podUID="659af87c-0f4c-4e8f-a977-c02e06e128af" containerName="cleanup" containerID="cri-o://f1ba71317e87520a6ec564340b8045dda20e5abd1ac4f09af0bf3d73d9131429" gracePeriod=30 Jul 08 18:16:00.023804 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:00.023782 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725575-vzdhd"] Jul 08 18:16:00.023994 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:00.023962 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-vzdhd" podUID="c4ffdc8e-61db-44b2-a8fa-f9472a262523" containerName="cleanup" containerID="cri-o://47cab145af4ebb34e719a5b7d5f15500c3c643459938f410cf1a5bdaaca39d01" gracePeriod=30 Jul 08 18:16:30.227485 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.227459 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7" Jul 08 18:16:30.252984 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.252949 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbcdm\" (UniqueName: \"kubernetes.io/projected/40999c7e-ada3-47b3-a6a0-cdd2c86d16c8-kube-api-access-mbcdm\") pod \"40999c7e-ada3-47b3-a6a0-cdd2c86d16c8\" (UID: \"40999c7e-ada3-47b3-a6a0-cdd2c86d16c8\") " Jul 08 18:16:30.256543 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.256510 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40999c7e-ada3-47b3-a6a0-cdd2c86d16c8-kube-api-access-mbcdm" (OuterVolumeSpecName: "kube-api-access-mbcdm") pod "40999c7e-ada3-47b3-a6a0-cdd2c86d16c8" (UID: "40999c7e-ada3-47b3-a6a0-cdd2c86d16c8"). InnerVolumeSpecName "kube-api-access-mbcdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:16:30.284041 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.284019 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-vzdhd" Jul 08 18:16:30.288361 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.288321 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q" Jul 08 18:16:30.354278 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.354231 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vbrd\" (UniqueName: \"kubernetes.io/projected/659af87c-0f4c-4e8f-a977-c02e06e128af-kube-api-access-4vbrd\") pod \"659af87c-0f4c-4e8f-a977-c02e06e128af\" (UID: \"659af87c-0f4c-4e8f-a977-c02e06e128af\") " Jul 08 18:16:30.354447 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.354334 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c92j9\" (UniqueName: \"kubernetes.io/projected/c4ffdc8e-61db-44b2-a8fa-f9472a262523-kube-api-access-c92j9\") pod \"c4ffdc8e-61db-44b2-a8fa-f9472a262523\" (UID: \"c4ffdc8e-61db-44b2-a8fa-f9472a262523\") " Jul 08 18:16:30.354627 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.354595 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbcdm\" (UniqueName: \"kubernetes.io/projected/40999c7e-ada3-47b3-a6a0-cdd2c86d16c8-kube-api-access-mbcdm\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:16:30.356234 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.356209 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659af87c-0f4c-4e8f-a977-c02e06e128af-kube-api-access-4vbrd" (OuterVolumeSpecName: "kube-api-access-4vbrd") pod "659af87c-0f4c-4e8f-a977-c02e06e128af" (UID: "659af87c-0f4c-4e8f-a977-c02e06e128af"). InnerVolumeSpecName "kube-api-access-4vbrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:16:30.356328 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.356269 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ffdc8e-61db-44b2-a8fa-f9472a262523-kube-api-access-c92j9" (OuterVolumeSpecName: "kube-api-access-c92j9") pod "c4ffdc8e-61db-44b2-a8fa-f9472a262523" (UID: "c4ffdc8e-61db-44b2-a8fa-f9472a262523"). InnerVolumeSpecName "kube-api-access-c92j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:16:30.455549 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.455515 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vbrd\" (UniqueName: \"kubernetes.io/projected/659af87c-0f4c-4e8f-a977-c02e06e128af-kube-api-access-4vbrd\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:16:30.455549 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.455545 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c92j9\" (UniqueName: \"kubernetes.io/projected/c4ffdc8e-61db-44b2-a8fa-f9472a262523-kube-api-access-c92j9\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:16:30.525909 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.525819 2584 generic.go:356] "Generic (PLEG): container finished" podID="659af87c-0f4c-4e8f-a977-c02e06e128af" containerID="f1ba71317e87520a6ec564340b8045dda20e5abd1ac4f09af0bf3d73d9131429" exitCode=137 Jul 08 18:16:30.525909 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.525861 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q" Jul 08 18:16:30.525909 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.525898 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q" event={"ID":"659af87c-0f4c-4e8f-a977-c02e06e128af","Type":"ContainerDied","Data":"f1ba71317e87520a6ec564340b8045dda20e5abd1ac4f09af0bf3d73d9131429"} Jul 08 18:16:30.526187 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.525947 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q" event={"ID":"659af87c-0f4c-4e8f-a977-c02e06e128af","Type":"ContainerDied","Data":"16e4ded29f5fcf6c4b0d41c69980bb386cd269d6e65c70d190b180803e85e80c"} Jul 08 18:16:30.526187 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.525970 2584 scope.go:117] "RemoveContainer" containerID="f1ba71317e87520a6ec564340b8045dda20e5abd1ac4f09af0bf3d73d9131429" Jul 08 18:16:30.527224 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.527200 2584 generic.go:356] "Generic (PLEG): container finished" podID="40999c7e-ada3-47b3-a6a0-cdd2c86d16c8" containerID="b900843ca779c5f5730413eb4f705892a4dd9d5aeda3aa201ae5a3ece3148870" exitCode=137 Jul 08 18:16:30.527334 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.527261 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7" Jul 08 18:16:30.527334 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.527267 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7" event={"ID":"40999c7e-ada3-47b3-a6a0-cdd2c86d16c8","Type":"ContainerDied","Data":"b900843ca779c5f5730413eb4f705892a4dd9d5aeda3aa201ae5a3ece3148870"} Jul 08 18:16:30.527334 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.527311 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7" event={"ID":"40999c7e-ada3-47b3-a6a0-cdd2c86d16c8","Type":"ContainerDied","Data":"d2410b6906af9e3e6411569e65b8b9ea77fe7d29fe739f5b3082dc195d722a54"} Jul 08 18:16:30.528680 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.528658 2584 generic.go:356] "Generic (PLEG): container finished" podID="c4ffdc8e-61db-44b2-a8fa-f9472a262523" containerID="47cab145af4ebb34e719a5b7d5f15500c3c643459938f410cf1a5bdaaca39d01" exitCode=137 Jul 08 18:16:30.528795 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.528686 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-vzdhd" event={"ID":"c4ffdc8e-61db-44b2-a8fa-f9472a262523","Type":"ContainerDied","Data":"47cab145af4ebb34e719a5b7d5f15500c3c643459938f410cf1a5bdaaca39d01"} Jul 08 18:16:30.528795 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.528704 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-vzdhd" Jul 08 18:16:30.528795 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.528732 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725575-vzdhd" event={"ID":"c4ffdc8e-61db-44b2-a8fa-f9472a262523","Type":"ContainerDied","Data":"44443a4b7fa5375531351e2b076c3a584f4edf6d45b7ba55e5182375140e89f3"} Jul 08 18:16:30.537854 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.537834 2584 scope.go:117] "RemoveContainer" containerID="f1ba71317e87520a6ec564340b8045dda20e5abd1ac4f09af0bf3d73d9131429" Jul 08 18:16:30.538119 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:16:30.538100 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1ba71317e87520a6ec564340b8045dda20e5abd1ac4f09af0bf3d73d9131429\": container with ID starting with f1ba71317e87520a6ec564340b8045dda20e5abd1ac4f09af0bf3d73d9131429 not found: ID does not exist" containerID="f1ba71317e87520a6ec564340b8045dda20e5abd1ac4f09af0bf3d73d9131429" Jul 08 18:16:30.538188 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.538127 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1ba71317e87520a6ec564340b8045dda20e5abd1ac4f09af0bf3d73d9131429"} err="failed to get container status \"f1ba71317e87520a6ec564340b8045dda20e5abd1ac4f09af0bf3d73d9131429\": rpc error: code = NotFound desc = could not find container \"f1ba71317e87520a6ec564340b8045dda20e5abd1ac4f09af0bf3d73d9131429\": container with ID starting with f1ba71317e87520a6ec564340b8045dda20e5abd1ac4f09af0bf3d73d9131429 not found: ID does not exist" Jul 08 18:16:30.538188 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.538143 2584 scope.go:117] "RemoveContainer" containerID="b900843ca779c5f5730413eb4f705892a4dd9d5aeda3aa201ae5a3ece3148870" Jul 08 18:16:30.547123 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.547100 2584 scope.go:117] "RemoveContainer" containerID="b900843ca779c5f5730413eb4f705892a4dd9d5aeda3aa201ae5a3ece3148870" Jul 08 18:16:30.547480 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:16:30.547459 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b900843ca779c5f5730413eb4f705892a4dd9d5aeda3aa201ae5a3ece3148870\": container with ID starting with b900843ca779c5f5730413eb4f705892a4dd9d5aeda3aa201ae5a3ece3148870 not found: ID does not exist" containerID="b900843ca779c5f5730413eb4f705892a4dd9d5aeda3aa201ae5a3ece3148870" Jul 08 18:16:30.547566 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.547487 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b900843ca779c5f5730413eb4f705892a4dd9d5aeda3aa201ae5a3ece3148870"} err="failed to get container status \"b900843ca779c5f5730413eb4f705892a4dd9d5aeda3aa201ae5a3ece3148870\": rpc error: code = NotFound desc = could not find container \"b900843ca779c5f5730413eb4f705892a4dd9d5aeda3aa201ae5a3ece3148870\": container with ID starting with b900843ca779c5f5730413eb4f705892a4dd9d5aeda3aa201ae5a3ece3148870 not found: ID does not exist" Jul 08 18:16:30.547566 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.547503 2584 scope.go:117] "RemoveContainer" containerID="47cab145af4ebb34e719a5b7d5f15500c3c643459938f410cf1a5bdaaca39d01" Jul 08 18:16:30.555461 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.555443 2584 scope.go:117] "RemoveContainer" containerID="47cab145af4ebb34e719a5b7d5f15500c3c643459938f410cf1a5bdaaca39d01" Jul 08 18:16:30.555713 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:16:30.555696 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47cab145af4ebb34e719a5b7d5f15500c3c643459938f410cf1a5bdaaca39d01\": container with ID starting with 47cab145af4ebb34e719a5b7d5f15500c3c643459938f410cf1a5bdaaca39d01 not found: ID does not exist" containerID="47cab145af4ebb34e719a5b7d5f15500c3c643459938f410cf1a5bdaaca39d01" Jul 08 18:16:30.555777 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.555720 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47cab145af4ebb34e719a5b7d5f15500c3c643459938f410cf1a5bdaaca39d01"} err="failed to get container status \"47cab145af4ebb34e719a5b7d5f15500c3c643459938f410cf1a5bdaaca39d01\": rpc error: code = NotFound desc = could not find container \"47cab145af4ebb34e719a5b7d5f15500c3c643459938f410cf1a5bdaaca39d01\": container with ID starting with 47cab145af4ebb34e719a5b7d5f15500c3c643459938f410cf1a5bdaaca39d01 not found: ID does not exist" Jul 08 18:16:30.560450 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.560426 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7"] Jul 08 18:16:30.567962 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.567933 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-1c14cf-29725575-wsdq7"] Jul 08 18:16:30.586489 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.586456 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q"] Jul 08 18:16:30.591159 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.591124 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-d1aec6-29725575-dqp4q"] Jul 08 18:16:30.605002 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.604969 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725575-vzdhd"] Jul 08 18:16:30.609648 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.609615 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725575-vzdhd"] Jul 08 18:16:30.957395 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.957363 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40999c7e-ada3-47b3-a6a0-cdd2c86d16c8" path="/var/lib/kubelet/pods/40999c7e-ada3-47b3-a6a0-cdd2c86d16c8/volumes" Jul 08 18:16:30.957721 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.957708 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="659af87c-0f4c-4e8f-a977-c02e06e128af" path="/var/lib/kubelet/pods/659af87c-0f4c-4e8f-a977-c02e06e128af/volumes" Jul 08 18:16:30.957998 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:16:30.957987 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4ffdc8e-61db-44b2-a8fa-f9472a262523" path="/var/lib/kubelet/pods/c4ffdc8e-61db-44b2-a8fa-f9472a262523/volumes" Jul 08 18:17:12.809891 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:12.809847 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:29.223435 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:29.223370 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:29.604140 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:29.604032 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:29.878044 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:29.878003 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:29.973528 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:29.973477 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:30.215571 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.215478 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz"] Jul 08 18:17:30.215739 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.215712 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" podUID="fac69956-a30b-4da8-b091-1fcc487d71ac" containerName="maas-api" containerID="cri-o://8973316fe7a595dea3e7dc1b85929dfdd525a59485c32ee78b00ffb814f699ca" gracePeriod=30 Jul 08 18:17:30.469057 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.468991 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" Jul 08 18:17:30.612092 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.612056 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fac69956-a30b-4da8-b091-1fcc487d71ac-openshift-service-ca\") pod \"fac69956-a30b-4da8-b091-1fcc487d71ac\" (UID: \"fac69956-a30b-4da8-b091-1fcc487d71ac\") " Jul 08 18:17:30.612305 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.612104 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bb2x\" (UniqueName: \"kubernetes.io/projected/fac69956-a30b-4da8-b091-1fcc487d71ac-kube-api-access-5bb2x\") pod \"fac69956-a30b-4da8-b091-1fcc487d71ac\" (UID: \"fac69956-a30b-4da8-b091-1fcc487d71ac\") " Jul 08 18:17:30.612305 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.612135 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fac69956-a30b-4da8-b091-1fcc487d71ac-maas-api-tls\") pod \"fac69956-a30b-4da8-b091-1fcc487d71ac\" (UID: \"fac69956-a30b-4da8-b091-1fcc487d71ac\") " Jul 08 18:17:30.612561 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.612527 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac69956-a30b-4da8-b091-1fcc487d71ac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fac69956-a30b-4da8-b091-1fcc487d71ac" (UID: "fac69956-a30b-4da8-b091-1fcc487d71ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:17:30.614290 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.614269 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fac69956-a30b-4da8-b091-1fcc487d71ac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fac69956-a30b-4da8-b091-1fcc487d71ac" (UID: "fac69956-a30b-4da8-b091-1fcc487d71ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:17:30.614396 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.614373 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac69956-a30b-4da8-b091-1fcc487d71ac-kube-api-access-5bb2x" (OuterVolumeSpecName: "kube-api-access-5bb2x") pod "fac69956-a30b-4da8-b091-1fcc487d71ac" (UID: "fac69956-a30b-4da8-b091-1fcc487d71ac"). InnerVolumeSpecName "kube-api-access-5bb2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:17:30.712970 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.712935 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fac69956-a30b-4da8-b091-1fcc487d71ac-maas-api-tls\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:17:30.712970 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.712962 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fac69956-a30b-4da8-b091-1fcc487d71ac-openshift-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:17:30.712970 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.712972 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bb2x\" (UniqueName: \"kubernetes.io/projected/fac69956-a30b-4da8-b091-1fcc487d71ac-kube-api-access-5bb2x\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:17:30.883649 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.883609 2584 generic.go:356] "Generic (PLEG): container finished" podID="fac69956-a30b-4da8-b091-1fcc487d71ac" containerID="8973316fe7a595dea3e7dc1b85929dfdd525a59485c32ee78b00ffb814f699ca" exitCode=0 Jul 08 18:17:30.883817 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.883654 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" Jul 08 18:17:30.883817 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.883689 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" event={"ID":"fac69956-a30b-4da8-b091-1fcc487d71ac","Type":"ContainerDied","Data":"8973316fe7a595dea3e7dc1b85929dfdd525a59485c32ee78b00ffb814f699ca"} Jul 08 18:17:30.883817 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.883739 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz" event={"ID":"fac69956-a30b-4da8-b091-1fcc487d71ac","Type":"ContainerDied","Data":"99997602d48c7f34b328fab75fbbd1036aa0483ac6639076ec4f0f895bbc3f62"} Jul 08 18:17:30.883817 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.883761 2584 scope.go:117] "RemoveContainer" containerID="8973316fe7a595dea3e7dc1b85929dfdd525a59485c32ee78b00ffb814f699ca" Jul 08 18:17:30.893136 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.893120 2584 scope.go:117] "RemoveContainer" containerID="8973316fe7a595dea3e7dc1b85929dfdd525a59485c32ee78b00ffb814f699ca" Jul 08 18:17:30.893414 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:17:30.893398 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8973316fe7a595dea3e7dc1b85929dfdd525a59485c32ee78b00ffb814f699ca\": container with ID starting with 8973316fe7a595dea3e7dc1b85929dfdd525a59485c32ee78b00ffb814f699ca not found: ID does not exist" containerID="8973316fe7a595dea3e7dc1b85929dfdd525a59485c32ee78b00ffb814f699ca" Jul 08 18:17:30.893485 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.893424 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8973316fe7a595dea3e7dc1b85929dfdd525a59485c32ee78b00ffb814f699ca"} err="failed to get container status \"8973316fe7a595dea3e7dc1b85929dfdd525a59485c32ee78b00ffb814f699ca\": rpc error: code = NotFound desc = could not find container \"8973316fe7a595dea3e7dc1b85929dfdd525a59485c32ee78b00ffb814f699ca\": container with ID starting with 8973316fe7a595dea3e7dc1b85929dfdd525a59485c32ee78b00ffb814f699ca not found: ID does not exist" Jul 08 18:17:30.910231 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.910204 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz"] Jul 08 18:17:30.920783 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.920762 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1c14cf-7cfd6c756c-49hnz"] Jul 08 18:17:30.956321 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:30.956290 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fac69956-a30b-4da8-b091-1fcc487d71ac" path="/var/lib/kubelet/pods/fac69956-a30b-4da8-b091-1fcc487d71ac/volumes" Jul 08 18:17:31.755436 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:31.755395 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:36.168975 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:36.168929 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:36.245707 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:36.245669 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:36.494698 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:36.494608 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:36.902095 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:36.902053 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:37.277522 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:37.277394 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:37.836084 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:37.836052 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw"] Jul 08 18:17:37.836344 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:37.836315 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" podUID="ff591338-f323-457a-95ae-41f146f5fdd2" containerName="maas-api" containerID="cri-o://cfe638c8e0910d3548f8448fefdad25c9f70ad46732149895b67a0f2084bc2b7" gracePeriod=30 Jul 08 18:17:38.086684 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:38.086618 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" Jul 08 18:17:38.181076 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:38.181047 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff591338-f323-457a-95ae-41f146f5fdd2-openshift-service-ca\") pod \"ff591338-f323-457a-95ae-41f146f5fdd2\" (UID: \"ff591338-f323-457a-95ae-41f146f5fdd2\") " Jul 08 18:17:38.181286 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:38.181087 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8tdr\" (UniqueName: \"kubernetes.io/projected/ff591338-f323-457a-95ae-41f146f5fdd2-kube-api-access-k8tdr\") pod \"ff591338-f323-457a-95ae-41f146f5fdd2\" (UID: \"ff591338-f323-457a-95ae-41f146f5fdd2\") " Jul 08 18:17:38.181286 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:38.181177 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff591338-f323-457a-95ae-41f146f5fdd2-maas-api-tls\") pod \"ff591338-f323-457a-95ae-41f146f5fdd2\" (UID: \"ff591338-f323-457a-95ae-41f146f5fdd2\") " Jul 08 18:17:38.181446 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:38.181416 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff591338-f323-457a-95ae-41f146f5fdd2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ff591338-f323-457a-95ae-41f146f5fdd2" (UID: "ff591338-f323-457a-95ae-41f146f5fdd2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:17:38.183222 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:38.183197 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff591338-f323-457a-95ae-41f146f5fdd2-kube-api-access-k8tdr" (OuterVolumeSpecName: "kube-api-access-k8tdr") pod "ff591338-f323-457a-95ae-41f146f5fdd2" (UID: "ff591338-f323-457a-95ae-41f146f5fdd2"). InnerVolumeSpecName "kube-api-access-k8tdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:17:38.183348 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:38.183315 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff591338-f323-457a-95ae-41f146f5fdd2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ff591338-f323-457a-95ae-41f146f5fdd2" (UID: "ff591338-f323-457a-95ae-41f146f5fdd2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 18:17:38.282851 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:38.282812 2584 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff591338-f323-457a-95ae-41f146f5fdd2-maas-api-tls\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:17:38.282851 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:38.282846 2584 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff591338-f323-457a-95ae-41f146f5fdd2-openshift-service-ca\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:17:38.282851 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:38.282856 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8tdr\" (UniqueName: \"kubernetes.io/projected/ff591338-f323-457a-95ae-41f146f5fdd2-kube-api-access-k8tdr\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:17:38.931962 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:38.931927 2584 generic.go:356] "Generic (PLEG): container finished" podID="ff591338-f323-457a-95ae-41f146f5fdd2" containerID="cfe638c8e0910d3548f8448fefdad25c9f70ad46732149895b67a0f2084bc2b7" exitCode=0 Jul 08 18:17:38.932134 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:38.931991 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" Jul 08 18:17:38.932134 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:38.932009 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" event={"ID":"ff591338-f323-457a-95ae-41f146f5fdd2","Type":"ContainerDied","Data":"cfe638c8e0910d3548f8448fefdad25c9f70ad46732149895b67a0f2084bc2b7"} Jul 08 18:17:38.932134 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:38.932066 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw" event={"ID":"ff591338-f323-457a-95ae-41f146f5fdd2","Type":"ContainerDied","Data":"3291b82f2aa149bc3d301dbe885f8ba855d37fc052d0cf319a6cae65d91f7986"} Jul 08 18:17:38.932134 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:38.932093 2584 scope.go:117] "RemoveContainer" containerID="cfe638c8e0910d3548f8448fefdad25c9f70ad46732149895b67a0f2084bc2b7" Jul 08 18:17:38.942011 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:38.941995 2584 scope.go:117] "RemoveContainer" containerID="cfe638c8e0910d3548f8448fefdad25c9f70ad46732149895b67a0f2084bc2b7" Jul 08 18:17:38.942289 ip-10-0-129-226 kubenswrapper[2584]: E0708 18:17:38.942269 2584 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfe638c8e0910d3548f8448fefdad25c9f70ad46732149895b67a0f2084bc2b7\": container with ID starting with cfe638c8e0910d3548f8448fefdad25c9f70ad46732149895b67a0f2084bc2b7 not found: ID does not exist" containerID="cfe638c8e0910d3548f8448fefdad25c9f70ad46732149895b67a0f2084bc2b7" Jul 08 18:17:38.942341 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:38.942299 2584 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfe638c8e0910d3548f8448fefdad25c9f70ad46732149895b67a0f2084bc2b7"} err="failed to get container status \"cfe638c8e0910d3548f8448fefdad25c9f70ad46732149895b67a0f2084bc2b7\": rpc error: code = NotFound desc = could not find container \"cfe638c8e0910d3548f8448fefdad25c9f70ad46732149895b67a0f2084bc2b7\": container with ID starting with cfe638c8e0910d3548f8448fefdad25c9f70ad46732149895b67a0f2084bc2b7 not found: ID does not exist" Jul 08 18:17:38.958193 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:38.958142 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw"] Jul 08 18:17:38.960402 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:38.960383 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-d1aec6-6684876cd6-jnjpw"] Jul 08 18:17:39.361428 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:39.361339 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:39.420158 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:39.420116 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:40.956592 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:40.956558 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff591338-f323-457a-95ae-41f146f5fdd2" path="/var/lib/kubelet/pods/ff591338-f323-457a-95ae-41f146f5fdd2/volumes" Jul 08 18:17:44.597058 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:44.597023 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:44.652338 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:44.652295 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:44.847777 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:44.847690 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj"] Jul 08 18:17:44.847934 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:44.847909 2584 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" podUID="ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" containerName="istio-proxy" containerID="cri-o://3432beecc819c12655850bc240396bb932e14be628510c4856aef93fef9b64ca" gracePeriod=30 Jul 08 18:17:44.864281 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:44.864215 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:45.643653 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:45.643621 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c44cb9c6-62sng_577df30a-813c-4063-bc7f-3c9d83edea32/authorino/0.log" Jul 08 18:17:50.000342 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.000306 2584 generic.go:356] "Generic (PLEG): container finished" podID="ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" containerID="3432beecc819c12655850bc240396bb932e14be628510c4856aef93fef9b64ca" exitCode=0 Jul 08 18:17:50.000810 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.000371 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" event={"ID":"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a","Type":"ContainerDied","Data":"3432beecc819c12655850bc240396bb932e14be628510c4856aef93fef9b64ca"} Jul 08 18:17:50.000810 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.000452 2584 scope.go:117] "RemoveContainer" containerID="31fe619dba40ae7cbef6510b40897ae019e96b5e39e4e2bb77615790cb681a85" Jul 08 18:17:50.099600 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.099575 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:17:50.194410 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.194366 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-workload-certs\") pod \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " Jul 08 18:17:50.194410 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.194415 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-credential-socket\") pod \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " Jul 08 18:17:50.194650 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.194434 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-podinfo\") pod \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " Jul 08 18:17:50.194650 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.194456 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-token\") pod \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " Jul 08 18:17:50.194650 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.194483 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-envoy\") pod \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " Jul 08 18:17:50.194650 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.194518 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-data\") pod \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " Jul 08 18:17:50.194650 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.194543 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-workload-socket\") pod \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " Jul 08 18:17:50.194650 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.194566 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istiod-ca-cert\") pod \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " Jul 08 18:17:50.194650 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.194591 2584 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nffj5\" (UniqueName: \"kubernetes.io/projected/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-kube-api-access-nffj5\") pod \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\" (UID: \"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a\") " Jul 08 18:17:50.195003 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.194692 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" (UID: "ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:17:50.195003 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.194769 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" (UID: "ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:17:50.195003 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.194859 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" (UID: "ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:17:50.195159 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.195029 2584 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-workload-certs\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:17:50.195159 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.195049 2584 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-credential-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:17:50.195159 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.195076 2584 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-workload-socket\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:17:50.195427 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.195403 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" (UID: "ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 18:17:50.195910 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.195879 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-data" (OuterVolumeSpecName: "istio-data") pod "ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" (UID: "ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:17:50.196858 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.196833 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-kube-api-access-nffj5" (OuterVolumeSpecName: "kube-api-access-nffj5") pod "ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" (UID: "ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a"). InnerVolumeSpecName "kube-api-access-nffj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:17:50.196973 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.196934 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" (UID: "ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 18:17:50.197146 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.197120 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-token" (OuterVolumeSpecName: "istio-token") pod "ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" (UID: "ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 18:17:50.197212 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.197116 2584 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" (UID: "ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 18:17:50.220129 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.220105 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-k56cv_8bc4bc8b-6633-48d4-942d-3eae3cca72b8/manager/0.log" Jul 08 18:17:50.296424 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.296391 2584 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istiod-ca-cert\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:17:50.296424 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.296423 2584 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nffj5\" (UniqueName: \"kubernetes.io/projected/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-kube-api-access-nffj5\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:17:50.296629 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.296435 2584 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-podinfo\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:17:50.296629 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.296447 2584 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-token\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:17:50.296629 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.296458 2584 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-envoy\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:17:50.296629 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.296469 2584 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a-istio-data\") on node \"ip-10-0-129-226.ec2.internal\" DevicePath \"\"" Jul 08 18:17:50.342190 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.342157 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-6fkzv_aba62f91-4709-4884-8435-74012fe550ea/manager/0.log" Jul 08 18:17:50.589656 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.589569 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vcspn_6be30b1e-b173-44b1-a2d3-39cd8183b9cc/server/0.log" Jul 08 18:17:50.708801 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.708769 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-btzq8_53eb9f89-11ea-48d4-9c9c-6720d0ae0950/manager/0.log" Jul 08 18:17:50.847346 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.847270 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dcf78d495-42ltt_1236d4aa-7255-4ce5-ac73-1bf9bb53806a/copy-manifests/0.log" Jul 08 18:17:50.871122 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:50.871083 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dcf78d495-42ltt_1236d4aa-7255-4ce5-ac73-1bf9bb53806a/manager/0.log" Jul 08 18:17:51.007478 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:51.007452 2584 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" Jul 08 18:17:51.008009 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:51.007488 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj" event={"ID":"ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a","Type":"ContainerDied","Data":"8bcaac66b13787404a666b489aead9e72ca4c88f2de48171f3723037e75af408"} Jul 08 18:17:51.008009 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:51.007544 2584 scope.go:117] "RemoveContainer" containerID="3432beecc819c12655850bc240396bb932e14be628510c4856aef93fef9b64ca" Jul 08 18:17:51.028633 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:51.028605 2584 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj"] Jul 08 18:17:51.033532 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:51.033510 2584 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-d1aec6-openshift-default-c5b774c7f-7qklj"] Jul 08 18:17:51.902779 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:51.902749 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25_e0ab5324-45ef-4724-a254-4bd8d0c11316/util/0.log" Jul 08 18:17:51.910605 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:51.910575 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25_e0ab5324-45ef-4724-a254-4bd8d0c11316/pull/0.log" Jul 08 18:17:51.918485 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:51.918462 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25_e0ab5324-45ef-4724-a254-4bd8d0c11316/extract/0.log" Jul 08 18:17:52.024097 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:52.024067 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7_03f69836-4f17-4951-865c-8998a1d10df1/util/0.log" Jul 08 18:17:52.039375 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:52.039343 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7_03f69836-4f17-4951-865c-8998a1d10df1/pull/0.log" Jul 08 18:17:52.057316 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:52.057225 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7_03f69836-4f17-4951-865c-8998a1d10df1/extract/0.log" Jul 08 18:17:52.169431 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:52.169353 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b_2f73576f-4b44-4102-9663-06af69ca2856/pull/0.log" Jul 08 18:17:52.177554 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:52.177533 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b_2f73576f-4b44-4102-9663-06af69ca2856/extract/0.log" Jul 08 18:17:52.184526 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:52.184504 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b_2f73576f-4b44-4102-9663-06af69ca2856/util/0.log" Jul 08 18:17:52.294699 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:52.294670 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26_017ec1dd-2a2b-47b4-bfc5-d2aa052fc186/util/0.log" Jul 08 18:17:52.303650 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:52.303622 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26_017ec1dd-2a2b-47b4-bfc5-d2aa052fc186/pull/0.log" Jul 08 18:17:52.311076 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:52.311049 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26_017ec1dd-2a2b-47b4-bfc5-d2aa052fc186/extract/0.log" Jul 08 18:17:52.431316 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:52.431205 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c44cb9c6-62sng_577df30a-813c-4063-bc7f-3c9d83edea32/authorino/0.log" Jul 08 18:17:52.558251 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:52.558197 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-ddxqk_a47d1f9b-ab0a-4af1-b9b6-3b8988203231/manager/0.log" Jul 08 18:17:52.679009 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:52.678980 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bcbfd_efa486df-ee78-4a9e-994a-44722b368f47/manager/0.log" Jul 08 18:17:52.791436 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:52.791363 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9js7v_2cab7921-a494-44d1-96fe-9f1bf1af5b32/kuadrant-console-plugin/0.log" Jul 08 18:17:52.913348 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:52.913321 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-94dzc_4a3e7aea-8da6-4392-a897-bcd5d475a640/registry-server/0.log" Jul 08 18:17:52.957009 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:52.956973 2584 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" path="/var/lib/kubelet/pods/ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a/volumes" Jul 08 18:17:53.037724 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:53.037697 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-7jdjn_8a936399-3688-4348-ad1d-28c3c02060f6/manager/0.log" Jul 08 18:17:53.274004 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:53.273972 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-rd5p5_5f2e7b50-7ca5-43d2-8c6b-44c13a9eb716/manager/0.log" Jul 08 18:17:53.942703 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:53.942671 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64959b45fc-pdgnb_7610607e-f0be-48c3-bdac-57396e5fb33e/kube-auth-proxy/0.log" Jul 08 18:17:54.403414 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:54.403379 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76b767f9f7-wwf7r_7811ee0a-b7e8-45d3-981f-4f81813aeae7/router/0.log" Jul 08 18:17:54.737829 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:54.737744 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-scctl_2f028844-9c43-410b-ba1b-cad906fb0553/main/0.log" Jul 08 18:17:58.860156 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:58.860118 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:58.960428 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:58.960397 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:59.206827 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.206792 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gdpqb/must-gather-rtfbw"] Jul 08 18:17:59.207083 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207050 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:59.207342 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207326 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fac69956-a30b-4da8-b091-1fcc487d71ac" containerName="maas-api" Jul 08 18:17:59.207405 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207344 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="fac69956-a30b-4da8-b091-1fcc487d71ac" containerName="maas-api" Jul 08 18:17:59.207405 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207354 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4ffdc8e-61db-44b2-a8fa-f9472a262523" containerName="cleanup" Jul 08 18:17:59.207405 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207368 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4ffdc8e-61db-44b2-a8fa-f9472a262523" containerName="cleanup" Jul 08 18:17:59.207405 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207384 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" containerName="istio-proxy" Jul 08 18:17:59.207405 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207390 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" containerName="istio-proxy" Jul 08 18:17:59.207405 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207397 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="659af87c-0f4c-4e8f-a977-c02e06e128af" containerName="cleanup" Jul 08 18:17:59.207405 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207404 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="659af87c-0f4c-4e8f-a977-c02e06e128af" containerName="cleanup" Jul 08 18:17:59.207610 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207420 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff591338-f323-457a-95ae-41f146f5fdd2" containerName="maas-api" Jul 08 18:17:59.207610 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207426 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff591338-f323-457a-95ae-41f146f5fdd2" containerName="maas-api" Jul 08 18:17:59.207610 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207434 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40999c7e-ada3-47b3-a6a0-cdd2c86d16c8" containerName="cleanup" Jul 08 18:17:59.207610 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207439 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="40999c7e-ada3-47b3-a6a0-cdd2c86d16c8" containerName="cleanup" Jul 08 18:17:59.207610 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207493 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" containerName="istio-proxy" Jul 08 18:17:59.207610 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207500 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" containerName="istio-proxy" Jul 08 18:17:59.207610 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207506 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="659af87c-0f4c-4e8f-a977-c02e06e128af" containerName="cleanup" Jul 08 18:17:59.207610 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207514 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="40999c7e-ada3-47b3-a6a0-cdd2c86d16c8" containerName="cleanup" Jul 08 18:17:59.207610 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207520 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4ffdc8e-61db-44b2-a8fa-f9472a262523" containerName="cleanup" Jul 08 18:17:59.207610 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207525 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="fac69956-a30b-4da8-b091-1fcc487d71ac" containerName="maas-api" Jul 08 18:17:59.207610 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207531 2584 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff591338-f323-457a-95ae-41f146f5fdd2" containerName="maas-api" Jul 08 18:17:59.207610 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207608 2584 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" containerName="istio-proxy" Jul 08 18:17:59.207610 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.207614 2584 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec5f4bc8-d83f-4f17-91f3-3cdce596fb0a" containerName="istio-proxy" Jul 08 18:17:59.212145 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.212129 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gdpqb/must-gather-rtfbw" Jul 08 18:17:59.214405 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.214388 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gdpqb\"/\"kube-root-ca.crt\"" Jul 08 18:17:59.215158 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.215143 2584 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gdpqb\"/\"default-dockercfg-tn67x\"" Jul 08 18:17:59.215254 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.215158 2584 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gdpqb\"/\"openshift-service-ca.crt\"" Jul 08 18:17:59.221711 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.221690 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gdpqb/must-gather-rtfbw"] Jul 08 18:17:59.262210 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.262176 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bd8552b3-c0bc-4dbf-bf42-bc82fc2a5755-must-gather-output\") pod \"must-gather-rtfbw\" (UID: \"bd8552b3-c0bc-4dbf-bf42-bc82fc2a5755\") " pod="openshift-must-gather-gdpqb/must-gather-rtfbw" Jul 08 18:17:59.262439 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.262228 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rm5f\" (UniqueName: \"kubernetes.io/projected/bd8552b3-c0bc-4dbf-bf42-bc82fc2a5755-kube-api-access-5rm5f\") pod \"must-gather-rtfbw\" (UID: \"bd8552b3-c0bc-4dbf-bf42-bc82fc2a5755\") " pod="openshift-must-gather-gdpqb/must-gather-rtfbw" Jul 08 18:17:59.283778 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.283746 2584 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" limit=1024 Jul 08 18:17:59.363608 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.363565 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rm5f\" (UniqueName: \"kubernetes.io/projected/bd8552b3-c0bc-4dbf-bf42-bc82fc2a5755-kube-api-access-5rm5f\") pod \"must-gather-rtfbw\" (UID: \"bd8552b3-c0bc-4dbf-bf42-bc82fc2a5755\") " pod="openshift-must-gather-gdpqb/must-gather-rtfbw" Jul 08 18:17:59.363763 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.363685 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bd8552b3-c0bc-4dbf-bf42-bc82fc2a5755-must-gather-output\") pod \"must-gather-rtfbw\" (UID: \"bd8552b3-c0bc-4dbf-bf42-bc82fc2a5755\") " pod="openshift-must-gather-gdpqb/must-gather-rtfbw" Jul 08 18:17:59.363972 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.363954 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bd8552b3-c0bc-4dbf-bf42-bc82fc2a5755-must-gather-output\") pod \"must-gather-rtfbw\" (UID: \"bd8552b3-c0bc-4dbf-bf42-bc82fc2a5755\") " pod="openshift-must-gather-gdpqb/must-gather-rtfbw" Jul 08 18:17:59.375779 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.375742 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rm5f\" (UniqueName: \"kubernetes.io/projected/bd8552b3-c0bc-4dbf-bf42-bc82fc2a5755-kube-api-access-5rm5f\") pod \"must-gather-rtfbw\" (UID: \"bd8552b3-c0bc-4dbf-bf42-bc82fc2a5755\") " pod="openshift-must-gather-gdpqb/must-gather-rtfbw" Jul 08 18:17:59.521596 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.521516 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gdpqb/must-gather-rtfbw" Jul 08 18:17:59.667515 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.667483 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gdpqb/must-gather-rtfbw"] Jul 08 18:17:59.669023 ip-10-0-129-226 kubenswrapper[2584]: W0708 18:17:59.668994 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd8552b3_c0bc_4dbf_bf42_bc82fc2a5755.slice/crio-7663c2c36e63052c3740f0b1b7301fe585a297f6c775cb1aaf51ecd2d07d3a46 WatchSource:0}: Error finding container 7663c2c36e63052c3740f0b1b7301fe585a297f6c775cb1aaf51ecd2d07d3a46: Status 404 returned error can't find the container with id 7663c2c36e63052c3740f0b1b7301fe585a297f6c775cb1aaf51ecd2d07d3a46 Jul 08 18:17:59.670933 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:17:59.670918 2584 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 18:18:00.058783 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:00.058750 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gdpqb/must-gather-rtfbw" event={"ID":"bd8552b3-c0bc-4dbf-bf42-bc82fc2a5755","Type":"ContainerStarted","Data":"7663c2c36e63052c3740f0b1b7301fe585a297f6c775cb1aaf51ecd2d07d3a46"} Jul 08 18:18:01.066394 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:01.066320 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gdpqb/must-gather-rtfbw" event={"ID":"bd8552b3-c0bc-4dbf-bf42-bc82fc2a5755","Type":"ContainerStarted","Data":"8e689bf02e0db8c53abeec11c25dda96828d155be8b1b0da2a6938e6e2837f2b"} Jul 08 18:18:02.075083 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:02.075034 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gdpqb/must-gather-rtfbw" event={"ID":"bd8552b3-c0bc-4dbf-bf42-bc82fc2a5755","Type":"ContainerStarted","Data":"6763444a2d59a14dc879e1abdcafb5f08bf9f822cd6b7508c73ad05680a38012"} Jul 08 18:18:02.093929 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:02.093882 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gdpqb/must-gather-rtfbw" podStartSLOduration=1.8709933749999998 podStartE2EDuration="3.093865757s" totalImagesPullingTime="1.222872382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 18:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:02.092709596 +0000 UTC m=+2357.776806874" watchObservedRunningTime="2026-07-08 18:18:02.093865757 +0000 UTC m=+2357.777963057" Jul 08 18:18:02.878354 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:02.878321 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gbq5c_ac3c6d61-29c8-4bc9-82d8-78177eb9045c/global-pull-secret-syncer/0.log" Jul 08 18:18:03.002601 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:03.002575 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bl792_970dfcb3-0c92-4a96-95c9-fe1db4c711d6/konnectivity-agent/0.log" Jul 08 18:18:03.149574 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:03.149482 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-226.ec2.internal_66beb382798cae7e66e45d75b5f0b9d7/haproxy/0.log" Jul 08 18:18:07.734297 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:07.734220 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25_e0ab5324-45ef-4724-a254-4bd8d0c11316/extract/0.log" Jul 08 18:18:07.766171 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:07.766135 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25_e0ab5324-45ef-4724-a254-4bd8d0c11316/util/0.log" Jul 08 18:18:07.799311 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:07.799280 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759qvg25_e0ab5324-45ef-4724-a254-4bd8d0c11316/pull/0.log" Jul 08 18:18:07.848919 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:07.848842 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7_03f69836-4f17-4951-865c-8998a1d10df1/extract/0.log" Jul 08 18:18:07.886550 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:07.886513 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7_03f69836-4f17-4951-865c-8998a1d10df1/util/0.log" Jul 08 18:18:07.921298 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:07.921265 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0bz5t7_03f69836-4f17-4951-865c-8998a1d10df1/pull/0.log" Jul 08 18:18:07.978607 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:07.978578 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b_2f73576f-4b44-4102-9663-06af69ca2856/extract/0.log" Jul 08 18:18:08.020036 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:08.019945 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b_2f73576f-4b44-4102-9663-06af69ca2856/util/0.log" Jul 08 18:18:08.061343 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:08.061292 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed739rw2b_2f73576f-4b44-4102-9663-06af69ca2856/pull/0.log" Jul 08 18:18:08.108280 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:08.108230 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26_017ec1dd-2a2b-47b4-bfc5-d2aa052fc186/extract/0.log" Jul 08 18:18:08.164811 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:08.164778 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26_017ec1dd-2a2b-47b4-bfc5-d2aa052fc186/util/0.log" Jul 08 18:18:08.201856 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:08.201825 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1mjr26_017ec1dd-2a2b-47b4-bfc5-d2aa052fc186/pull/0.log" Jul 08 18:18:08.704178 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:08.704133 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5c44cb9c6-62sng_577df30a-813c-4063-bc7f-3c9d83edea32/authorino/0.log" Jul 08 18:18:08.740014 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:08.739979 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-ddxqk_a47d1f9b-ab0a-4af1-b9b6-3b8988203231/manager/0.log" Jul 08 18:18:08.785253 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:08.785202 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bcbfd_efa486df-ee78-4a9e-994a-44722b368f47/manager/0.log" Jul 08 18:18:08.863905 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:08.863874 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9js7v_2cab7921-a494-44d1-96fe-9f1bf1af5b32/kuadrant-console-plugin/0.log" Jul 08 18:18:08.952342 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:08.952308 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-94dzc_4a3e7aea-8da6-4392-a897-bcd5d475a640/registry-server/0.log" Jul 08 18:18:09.130486 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:09.129900 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-7jdjn_8a936399-3688-4348-ad1d-28c3c02060f6/manager/0.log" Jul 08 18:18:09.275263 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:09.275179 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-rd5p5_5f2e7b50-7ca5-43d2-8c6b-44c13a9eb716/manager/0.log" Jul 08 18:18:11.273802 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:11.273773 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-57fr6_e2388ecf-b94e-4738-9818-4ab78c6d1c80/cluster-monitoring-operator/0.log" Jul 08 18:18:11.716013 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:11.715972 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wnhc8_a1e0969a-1dab-4d4d-b4b2-f3d30910bc69/node-exporter/0.log" Jul 08 18:18:11.749144 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:11.749115 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wnhc8_a1e0969a-1dab-4d4d-b4b2-f3d30910bc69/kube-rbac-proxy/0.log" Jul 08 18:18:11.778020 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:11.777985 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wnhc8_a1e0969a-1dab-4d4d-b4b2-f3d30910bc69/init-textfile/0.log" Jul 08 18:18:12.673876 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:12.673825 2584 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp"] Jul 08 18:18:12.688355 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:12.688311 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" Jul 08 18:18:12.693313 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:12.690103 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp"] Jul 08 18:18:12.709679 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:12.709643 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec5f3fc0-35af-4e56-8c76-fd25b3746b0f-sys\") pod \"perf-node-gather-daemonset-hvxvp\" (UID: \"ec5f3fc0-35af-4e56-8c76-fd25b3746b0f\") " pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" Jul 08 18:18:12.709822 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:12.709693 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec5f3fc0-35af-4e56-8c76-fd25b3746b0f-lib-modules\") pod \"perf-node-gather-daemonset-hvxvp\" (UID: \"ec5f3fc0-35af-4e56-8c76-fd25b3746b0f\") " pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" Jul 08 18:18:12.709822 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:12.709731 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ec5f3fc0-35af-4e56-8c76-fd25b3746b0f-podres\") pod \"perf-node-gather-daemonset-hvxvp\" (UID: \"ec5f3fc0-35af-4e56-8c76-fd25b3746b0f\") " pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" Jul 08 18:18:12.709822 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:12.709795 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ec5f3fc0-35af-4e56-8c76-fd25b3746b0f-proc\") pod \"perf-node-gather-daemonset-hvxvp\" (UID: \"ec5f3fc0-35af-4e56-8c76-fd25b3746b0f\") " pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" Jul 08 18:18:12.709993 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:12.709839 2584 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sx89\" (UniqueName: \"kubernetes.io/projected/ec5f3fc0-35af-4e56-8c76-fd25b3746b0f-kube-api-access-7sx89\") pod \"perf-node-gather-daemonset-hvxvp\" (UID: \"ec5f3fc0-35af-4e56-8c76-fd25b3746b0f\") " pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" Jul 08 18:18:12.811282 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:12.811231 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec5f3fc0-35af-4e56-8c76-fd25b3746b0f-sys\") pod \"perf-node-gather-daemonset-hvxvp\" (UID: \"ec5f3fc0-35af-4e56-8c76-fd25b3746b0f\") " pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" Jul 08 18:18:12.811282 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:12.811287 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec5f3fc0-35af-4e56-8c76-fd25b3746b0f-lib-modules\") pod \"perf-node-gather-daemonset-hvxvp\" (UID: \"ec5f3fc0-35af-4e56-8c76-fd25b3746b0f\") " pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" Jul 08 18:18:12.811527 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:12.811313 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ec5f3fc0-35af-4e56-8c76-fd25b3746b0f-podres\") pod \"perf-node-gather-daemonset-hvxvp\" (UID: \"ec5f3fc0-35af-4e56-8c76-fd25b3746b0f\") " pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" Jul 08 18:18:12.811527 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:12.811347 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec5f3fc0-35af-4e56-8c76-fd25b3746b0f-sys\") pod \"perf-node-gather-daemonset-hvxvp\" (UID: \"ec5f3fc0-35af-4e56-8c76-fd25b3746b0f\") " pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" Jul 08 18:18:12.811527 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:12.811354 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ec5f3fc0-35af-4e56-8c76-fd25b3746b0f-proc\") pod \"perf-node-gather-daemonset-hvxvp\" (UID: \"ec5f3fc0-35af-4e56-8c76-fd25b3746b0f\") " pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" Jul 08 18:18:12.811527 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:12.811406 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ec5f3fc0-35af-4e56-8c76-fd25b3746b0f-proc\") pod \"perf-node-gather-daemonset-hvxvp\" (UID: \"ec5f3fc0-35af-4e56-8c76-fd25b3746b0f\") " pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" Jul 08 18:18:12.811527 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:12.811439 2584 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sx89\" (UniqueName: \"kubernetes.io/projected/ec5f3fc0-35af-4e56-8c76-fd25b3746b0f-kube-api-access-7sx89\") pod \"perf-node-gather-daemonset-hvxvp\" (UID: \"ec5f3fc0-35af-4e56-8c76-fd25b3746b0f\") " pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" Jul 08 18:18:12.811527 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:12.811449 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ec5f3fc0-35af-4e56-8c76-fd25b3746b0f-podres\") pod \"perf-node-gather-daemonset-hvxvp\" (UID: \"ec5f3fc0-35af-4e56-8c76-fd25b3746b0f\") " pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" Jul 08 18:18:12.811527 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:12.811462 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec5f3fc0-35af-4e56-8c76-fd25b3746b0f-lib-modules\") pod \"perf-node-gather-daemonset-hvxvp\" (UID: \"ec5f3fc0-35af-4e56-8c76-fd25b3746b0f\") " pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" Jul 08 18:18:12.821351 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:12.821328 2584 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sx89\" (UniqueName: \"kubernetes.io/projected/ec5f3fc0-35af-4e56-8c76-fd25b3746b0f-kube-api-access-7sx89\") pod \"perf-node-gather-daemonset-hvxvp\" (UID: \"ec5f3fc0-35af-4e56-8c76-fd25b3746b0f\") " pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" Jul 08 18:18:13.006263 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:13.006147 2584 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" Jul 08 18:18:13.202215 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:13.202167 2584 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp"] Jul 08 18:18:13.202465 ip-10-0-129-226 kubenswrapper[2584]: W0708 18:18:13.202430 2584 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podec5f3fc0_35af_4e56_8c76_fd25b3746b0f.slice/crio-69e24002fc02d9ca6e06d7a15e6abc3db5ea0dd66170b03ddfca42afc40b6515 WatchSource:0}: Error finding container 69e24002fc02d9ca6e06d7a15e6abc3db5ea0dd66170b03ddfca42afc40b6515: Status 404 returned error can't find the container with id 69e24002fc02d9ca6e06d7a15e6abc3db5ea0dd66170b03ddfca42afc40b6515 Jul 08 18:18:14.191777 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:14.191737 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" event={"ID":"ec5f3fc0-35af-4e56-8c76-fd25b3746b0f","Type":"ContainerStarted","Data":"442e00b487171d3a96599f1537b9c48b8a406a45730ceb28f2f15c935bb7a8cf"} Jul 08 18:18:14.192277 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:14.191799 2584 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" event={"ID":"ec5f3fc0-35af-4e56-8c76-fd25b3746b0f","Type":"ContainerStarted","Data":"69e24002fc02d9ca6e06d7a15e6abc3db5ea0dd66170b03ddfca42afc40b6515"} Jul 08 18:18:14.192277 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:14.191900 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" Jul 08 18:18:14.217028 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:14.216950 2584 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" podStartSLOduration=2.216928467 podStartE2EDuration="2.216928467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 18:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 18:18:14.21489541 +0000 UTC m=+2369.898992692" watchObservedRunningTime="2026-07-08 18:18:14.216928467 +0000 UTC m=+2369.901025745" Jul 08 18:18:14.452077 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:14.451980 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/1.log" Jul 08 18:18:14.461034 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:14.460994 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-sbh59_47839b2a-54b6-4723-bba7-f43ef243c1b1/console-operator/2.log" Jul 08 18:18:15.156620 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:15.156587 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-vtf7x_dc218f8b-426c-4d44-a0b5-09f65c1f2747/download-server/0.log" Jul 08 18:18:15.827616 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:15.827568 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-h6pfl_d9f0ea1a-0f7d-496d-bdc4-b0cf946ba514/volume-data-source-validator/0.log" Jul 08 18:18:17.255698 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:17.255666 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rn2rl_e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60/dns/0.log" Jul 08 18:18:17.294639 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:17.294607 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rn2rl_e8f6ff84-c9d3-4d4a-9c7f-7eda26065d60/kube-rbac-proxy/0.log" Jul 08 18:18:17.478256 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:17.478216 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8h4h6_e8468fb3-5642-43d6-95e5-c524fb532087/dns-node-resolver/0.log" Jul 08 18:18:18.211728 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:18.211693 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rhlxp_747083d6-492d-413c-99e7-364aca1c1cee/node-ca/0.log" Jul 08 18:18:19.598234 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:19.598201 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64959b45fc-pdgnb_7610607e-f0be-48c3-bdac-57396e5fb33e/kube-auth-proxy/0.log" Jul 08 18:18:19.833326 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:19.833296 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76b767f9f7-wwf7r_7811ee0a-b7e8-45d3-981f-4f81813aeae7/router/0.log" Jul 08 18:18:20.211208 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:20.210902 2584 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gdpqb/perf-node-gather-daemonset-hvxvp" Jul 08 18:18:20.671511 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:20.671469 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hfbxp_ce741a8f-ef63-4a40-a144-a4ce45e2e649/serve-healthcheck-canary/0.log" Jul 08 18:18:21.370829 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:21.370799 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-fqtgz_05b7e47c-d931-46f1-a21d-fcb853e77030/insights-operator/1.log" Jul 08 18:18:21.372508 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:21.372480 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-fqtgz_05b7e47c-d931-46f1-a21d-fcb853e77030/insights-operator/0.log" Jul 08 18:18:21.520819 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:21.520791 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9lmjs_c11cd9b6-1207-4c05-ac75-13396ce41e35/kube-rbac-proxy/0.log" Jul 08 18:18:21.561192 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:21.561123 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9lmjs_c11cd9b6-1207-4c05-ac75-13396ce41e35/exporter/0.log" Jul 08 18:18:21.599040 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:21.598958 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9lmjs_c11cd9b6-1207-4c05-ac75-13396ce41e35/extractor/0.log" Jul 08 18:18:23.437700 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:23.437668 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64959b45fc-pdgnb_7610607e-f0be-48c3-bdac-57396e5fb33e/kube-auth-proxy/0.log" Jul 08 18:18:23.624716 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:23.624642 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76b767f9f7-wwf7r_7811ee0a-b7e8-45d3-981f-4f81813aeae7/router/0.log" Jul 08 18:18:24.371037 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:24.371005 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-k56cv_8bc4bc8b-6633-48d4-942d-3eae3cca72b8/manager/0.log" Jul 08 18:18:24.583677 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:24.583644 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-6fkzv_aba62f91-4709-4884-8435-74012fe550ea/manager/0.log" Jul 08 18:18:24.774040 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:24.773944 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-vcspn_6be30b1e-b173-44b1-a2d3-39cd8183b9cc/server/0.log" Jul 08 18:18:24.888881 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:24.888835 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-btzq8_53eb9f89-11ea-48d4-9c9c-6720d0ae0950/manager/0.log" Jul 08 18:18:25.010849 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:25.007804 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dcf78d495-42ltt_1236d4aa-7255-4ce5-ac73-1bf9bb53806a/manager/0.log" Jul 08 18:18:25.042671 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:25.042596 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6dcf78d495-42ltt_1236d4aa-7255-4ce5-ac73-1bf9bb53806a/copy-manifests/0.log" Jul 08 18:18:25.096879 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:25.096848 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-64959b45fc-pdgnb_7610607e-f0be-48c3-bdac-57396e5fb33e/kube-auth-proxy/0.log" Jul 08 18:18:25.300879 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:25.300802 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-76b767f9f7-wwf7r_7811ee0a-b7e8-45d3-981f-4f81813aeae7/router/0.log" Jul 08 18:18:26.540528 ip-10-0-129-226 kubenswrapper[2584]: I0708 18:18:26.540467 2584 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-8xr9f_89020466-4f77-4a18-8f7c-a6570a2640b8/openshift-lws-operator/0.log"