Jul 10 18:58:31.535494 ip-10-0-130-74 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 10 18:58:31.535507 ip-10-0-130-74 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 10 18:58:31.535517 ip-10-0-130-74 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 10 18:58:31.535903 ip-10-0-130-74 systemd[1]: Failed to start Kubernetes Kubelet. Jul 10 18:58:41.613733 ip-10-0-130-74 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 10 18:58:41.613749 ip-10-0-130-74 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3cacb2f23c2d419f9b643a00048a7fa7 -- Jul 10 19:00:33.259312 ip-10-0-130-74 systemd[1]: Starting Kubernetes Kubelet... Jul 10 19:00:33.726966 ip-10-0-130-74 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 19:00:33.726966 ip-10-0-130-74 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 19:00:33.726966 ip-10-0-130-74 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 19:00:33.726966 ip-10-0-130-74 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 19:00:33.728747 ip-10-0-130-74 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 19:00:33.728877 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.728796 2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 19:00:33.738667 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738636 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 19:00:33.738667 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738660 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 19:00:33.738667 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738664 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 19:00:33.738667 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738667 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 19:00:33.738667 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738669 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 19:00:33.738667 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738672 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 19:00:33.738667 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738676 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 19:00:33.738667 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738678 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 19:00:33.738963 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738681 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 19:00:33.738963 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738684 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 19:00:33.738963 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738687 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 19:00:33.738963 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738698 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 19:00:33.738963 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738702 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 19:00:33.738963 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738705 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 19:00:33.738963 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738707 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 19:00:33.738963 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738710 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 19:00:33.738963 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738712 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 19:00:33.738963 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738715 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 19:00:33.738963 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738718 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 19:00:33.738963 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738721 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 19:00:33.738963 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738724 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 19:00:33.738963 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738729 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 19:00:33.738963 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738734 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 19:00:33.738963 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738737 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 19:00:33.738963 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738740 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 19:00:33.738963 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738743 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 19:00:33.738963 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738746 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738749 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738752 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738755 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738757 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738761 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738763 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738766 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738768 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738771 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738773 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738776 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738778 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738780 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738783 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738786 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738789 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738792 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738796 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738798 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 19:00:33.739411 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738801 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738803 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738806 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738808 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738811 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738813 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738817 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738820 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738823 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738826 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738828 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738831 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738833 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738836 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738839 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738841 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738843 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738846 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738849 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738851 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 19:00:33.739925 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738854 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738856 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738859 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738861 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738864 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738866 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738868 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738873 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738876 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738878 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738881 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738884 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738887 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738889 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738892 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738895 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738897 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738900 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738903 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738905 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 19:00:33.740418 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738908 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738910 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738913 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738915 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738918 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738921 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738923 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738925 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738928 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738930 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738949 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738953 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.738959 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739706 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739713 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739716 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739720 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739722 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739725 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739728 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 19:00:33.740893 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739732 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 19:00:33.741386 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739735 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 19:00:33.741386 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739738 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 19:00:33.741386 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739741 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 19:00:33.741386 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739743 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 19:00:33.741386 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739746 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 19:00:33.741386 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739749 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 19:00:33.741386 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739752 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 19:00:33.741386 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739754 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 19:00:33.741386 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739757 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 19:00:33.741386 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739765 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 19:00:33.741386 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739767 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 19:00:33.741386 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739770 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 19:00:33.741386 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739772 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 19:00:33.741386 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739776 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 19:00:33.741386 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739779 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 19:00:33.741386 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739781 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 19:00:33.741386 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739785 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 19:00:33.741386 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739789 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 19:00:33.741386 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739792 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739794 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739797 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739799 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739802 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739804 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739807 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739809 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739812 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739814 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739819 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739822 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739826 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739830 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739833 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739835 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739838 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739841 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739843 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739846 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 19:00:33.741844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739848 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739851 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739853 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739856 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739859 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739862 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739864 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739867 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739869 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739872 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739874 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739877 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739879 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739881 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739884 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739886 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739889 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739892 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739895 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739897 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 19:00:33.742359 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739899 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739902 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739904 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739907 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739909 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739912 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739914 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739917 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739919 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739922 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739925 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739929 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739948 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739951 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739954 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739957 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739959 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739962 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739966 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739968 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 19:00:33.742832 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739971 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 19:00:33.743331 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739974 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 19:00:33.743331 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739976 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 19:00:33.743331 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739979 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 19:00:33.743331 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739981 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 19:00:33.743331 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739984 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 19:00:33.743331 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739987 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 19:00:33.743331 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739989 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 19:00:33.743331 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739992 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 19:00:33.743331 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739994 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 19:00:33.743331 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739997 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 19:00:33.743331 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.739999 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 19:00:33.743331 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740002 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 19:00:33.743331 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740409 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 19:00:33.743331 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740414 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 19:00:33.743331 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740417 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 19:00:33.743331 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740420 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 19:00:33.743331 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740422 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 19:00:33.743331 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740425 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 19:00:33.743331 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740429 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740432 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740434 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740437 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740439 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740443 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740445 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740448 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740450 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740453 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740456 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740459 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740462 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740465 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740467 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740469 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740472 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740474 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740477 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740479 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 19:00:33.743816 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740482 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 19:00:33.744312 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740484 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 19:00:33.744312 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740487 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 19:00:33.744312 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740491 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 19:00:33.744312 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740494 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 19:00:33.744312 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740498 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 19:00:33.744312 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740500 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 19:00:33.744312 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740504 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 19:00:33.744312 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740507 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 19:00:33.744312 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740510 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 19:00:33.744312 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740512 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 19:00:33.744312 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740515 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 19:00:33.744312 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740517 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 19:00:33.744312 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740520 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 19:00:33.744312 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740522 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 19:00:33.744312 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740525 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 19:00:33.744312 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740527 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 19:00:33.744312 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740530 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 19:00:33.744312 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740532 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 19:00:33.744312 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740535 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 19:00:33.744780 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740537 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 19:00:33.744780 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740539 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 19:00:33.744780 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740542 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 19:00:33.744780 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740545 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 19:00:33.744780 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740548 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 19:00:33.744780 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740550 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 19:00:33.744780 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740553 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 19:00:33.744780 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740557 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 19:00:33.744780 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740560 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 19:00:33.744780 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740562 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 19:00:33.744780 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740564 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 19:00:33.744780 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740567 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 19:00:33.744780 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740569 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 19:00:33.744780 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740572 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 19:00:33.744780 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740574 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 19:00:33.744780 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740576 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 19:00:33.744780 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740579 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 19:00:33.744780 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740582 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 19:00:33.744780 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740584 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740587 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740589 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740592 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740594 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740597 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740599 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740601 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740604 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740607 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740609 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740612 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740614 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740617 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740619 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740622 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740624 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740627 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740633 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740635 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 19:00:33.745335 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740638 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740641 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740643 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740645 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740648 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740650 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740653 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740668 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740671 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740674 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740677 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740680 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740683 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740685 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.740688 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740764 2561 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740772 2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740780 2561 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740784 2561 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740788 2561 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740791 2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 19:00:33.745837 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740796 2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740800 2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740803 2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740806 2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740809 2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740812 2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740816 2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740820 2561 flags.go:64] FLAG: --cgroup-root="" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740823 2561 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740826 2561 flags.go:64] FLAG: --client-ca-file="" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740829 2561 flags.go:64] FLAG: --cloud-provider="external" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740832 2561 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740836 2561 flags.go:64] FLAG: --cluster-domain="" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740839 2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740842 2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740845 2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740848 2561 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740852 2561 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740856 2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740859 2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740862 2561 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740864 2561 flags.go:64] FLAG: --contention-profiling="false" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740868 2561 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740870 2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 19:00:33.746365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740873 2561 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740876 2561 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740880 2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740883 2561 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740886 2561 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740889 2561 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740891 2561 flags.go:64] FLAG: --enable-server="true" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740895 2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740899 2561 flags.go:64] FLAG: --event-burst="100" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740903 2561 flags.go:64] FLAG: --event-qps="50" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740906 2561 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740909 2561 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740911 2561 flags.go:64] FLAG: --eviction-hard="" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740915 2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740918 2561 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740924 2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740927 2561 flags.go:64] FLAG: --eviction-soft="" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740930 2561 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740948 2561 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740952 2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740955 2561 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740958 2561 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740961 2561 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740964 2561 flags.go:64] FLAG: --feature-gates="" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740967 2561 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 19:00:33.746980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740970 2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740974 2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740977 2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740980 2561 flags.go:64] FLAG: --healthz-port="10248" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740982 2561 flags.go:64] FLAG: --help="false" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740985 2561 flags.go:64] FLAG: --hostname-override="ip-10-0-130-74.ec2.internal" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740988 2561 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740991 2561 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740994 2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.740997 2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741001 2561 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741004 2561 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741007 2561 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741009 2561 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741012 2561 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741015 2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741018 2561 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741021 2561 flags.go:64] FLAG: --kube-reserved="" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741023 2561 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741026 2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741029 2561 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741031 2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741035 2561 flags.go:64] FLAG: --lock-file="" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741038 2561 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 19:00:33.747601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741041 2561 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741043 2561 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741049 2561 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741052 2561 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741054 2561 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741057 2561 flags.go:64] FLAG: --logging-format="text" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741060 2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741063 2561 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741065 2561 flags.go:64] FLAG: --manifest-url="" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741068 2561 flags.go:64] FLAG: --manifest-url-header="" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741073 2561 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741076 2561 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741079 2561 flags.go:64] FLAG: --max-pods="110" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741082 2561 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741085 2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741088 2561 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741091 2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741093 2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741096 2561 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741099 2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741106 2561 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741109 2561 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741112 2561 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 19:00:33.748267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741115 2561 flags.go:64] FLAG: --pod-cidr="" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741118 2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741123 2561 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741125 2561 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741128 2561 flags.go:64] FLAG: --pods-per-core="0" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741131 2561 flags.go:64] FLAG: --port="10250" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741134 2561 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741138 2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07ad645b168bbbfce" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741141 2561 flags.go:64] FLAG: --qos-reserved="" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741144 2561 flags.go:64] FLAG: --read-only-port="10255" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741147 2561 flags.go:64] FLAG: --register-node="true" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741150 2561 flags.go:64] FLAG: --register-with-taints="" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741153 2561 flags.go:64] FLAG: --registry-burst="10" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741157 2561 flags.go:64] FLAG: --registry-qps="5" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741159 2561 flags.go:64] FLAG: --reserved-cpus="" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741162 2561 flags.go:64] FLAG: --reserved-memory="" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741165 2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741168 2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741171 2561 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741174 2561 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741177 2561 flags.go:64] FLAG: --runonce="false" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741179 2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741182 2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741185 2561 flags.go:64] FLAG: --seccomp-default="false" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741188 2561 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 19:00:33.748825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741190 2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741193 2561 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741196 2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741199 2561 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741201 2561 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741204 2561 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741207 2561 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741209 2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741212 2561 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741222 2561 flags.go:64] FLAG: --system-cgroups="" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741225 2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741231 2561 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741233 2561 flags.go:64] FLAG: --tls-cert-file="" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741236 2561 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741241 2561 flags.go:64] FLAG: --tls-min-version="" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741244 2561 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741246 2561 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741249 2561 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741252 2561 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741254 2561 flags.go:64] FLAG: --v="2" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741258 2561 flags.go:64] FLAG: --version="false" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741263 2561 flags.go:64] FLAG: --vmodule="" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741267 2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.741270 2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741405 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 19:00:33.749459 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741409 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741412 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741415 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741418 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741421 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741423 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741426 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741428 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741431 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741434 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741437 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741440 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741443 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741445 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741448 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741451 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741453 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741455 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741458 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741460 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 19:00:33.750116 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741463 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741466 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741469 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741471 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741474 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741476 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741479 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741481 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741484 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741488 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741490 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741493 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741495 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741500 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741502 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741505 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741507 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741510 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741512 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741515 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 19:00:33.750599 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741518 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 19:00:33.751106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741522 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 19:00:33.751106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741525 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 19:00:33.751106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741528 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 19:00:33.751106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741530 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 19:00:33.751106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741533 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 19:00:33.751106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741536 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 19:00:33.751106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741538 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 19:00:33.751106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741541 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 19:00:33.751106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741544 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 19:00:33.751106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741548 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 19:00:33.751106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741551 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 19:00:33.751106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741554 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 19:00:33.751106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741558 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 19:00:33.751106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741560 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 19:00:33.751106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741563 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 19:00:33.751106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741566 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 19:00:33.751106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741569 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 10 19:00:33.751106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741571 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 19:00:33.751106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741574 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741576 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741579 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741582 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741584 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741587 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741590 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741593 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741595 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741598 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741600 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741603 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741605 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741608 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741610 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741614 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741617 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741620 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741622 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741625 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 19:00:33.751587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741627 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 19:00:33.752177 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741630 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 19:00:33.752177 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741632 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 19:00:33.752177 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741635 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 19:00:33.752177 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741637 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 19:00:33.752177 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741640 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 19:00:33.752177 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741643 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 19:00:33.752177 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741646 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 19:00:33.752177 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741649 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 19:00:33.752177 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741651 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 19:00:33.752177 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741654 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 19:00:33.752177 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741656 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 19:00:33.752177 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741659 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 19:00:33.752177 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741661 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 19:00:33.752177 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741664 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 19:00:33.752177 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741666 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 19:00:33.752177 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741669 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 19:00:33.752177 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741672 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 19:00:33.752177 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.741675 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 19:00:33.752617 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.742671 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 19:00:33.752617 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.749509 2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 19:00:33.752617 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.749527 2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 19:00:33.752617 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.751144 2561 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 19:00:33.752617 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.751152 2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 19:00:33.752617 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751218 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 19:00:33.752617 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751224 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 19:00:33.752617 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751228 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 19:00:33.752617 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751232 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 19:00:33.752617 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751235 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 19:00:33.752617 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751238 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 19:00:33.752617 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751241 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 19:00:33.752617 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751243 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 19:00:33.752617 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751246 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751249 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751251 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751254 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751257 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751259 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751261 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751264 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751266 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751269 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751272 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751276 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751280 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751283 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751286 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751290 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751293 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751296 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751299 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751302 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 19:00:33.752980 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751305 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751308 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751311 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751313 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751316 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751318 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751321 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751324 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751326 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751329 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751331 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751334 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751336 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751339 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751341 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751344 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751347 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751349 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751351 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751354 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 19:00:33.753480 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751356 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751359 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751362 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751365 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751368 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751371 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751373 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751375 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751378 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751381 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751383 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751386 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751389 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751392 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751394 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751397 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751400 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751402 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751404 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751407 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 19:00:33.753971 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751409 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751411 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751414 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751416 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751419 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751421 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751424 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751426 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751429 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751431 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751434 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751436 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751438 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751441 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751443 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751446 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751449 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751451 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751454 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751456 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 19:00:33.754536 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751459 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 19:00:33.755059 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751461 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 19:00:33.755059 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751464 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 19:00:33.755059 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751467 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 19:00:33.755059 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751470 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 19:00:33.755059 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751472 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 19:00:33.755059 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751475 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 19:00:33.755059 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751477 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 19:00:33.755059 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751480 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 19:00:33.755059 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751484 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 19:00:33.755059 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751487 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 19:00:33.755059 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.751492 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 19:00:33.755059 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751625 2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 19:00:33.755059 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751630 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 19:00:33.755059 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751633 2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 19:00:33.755059 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751635 2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 19:00:33.755430 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751638 2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 19:00:33.755430 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751641 2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 19:00:33.755430 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751644 2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 19:00:33.755430 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751646 2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 19:00:33.755430 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751649 2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 19:00:33.755430 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751651 2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 19:00:33.755430 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751654 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 19:00:33.755430 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751656 2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 19:00:33.755430 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751659 2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 19:00:33.755430 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751661 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 19:00:33.755430 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751664 2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 19:00:33.755430 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751666 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 19:00:33.755430 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751669 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 19:00:33.755430 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751671 2561 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 19:00:33.755430 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751674 2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 19:00:33.755430 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751677 2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 19:00:33.755430 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751679 2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 19:00:33.755430 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751682 2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 19:00:33.755430 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751686 2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 19:00:33.755900 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751690 2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 19:00:33.755900 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751693 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 19:00:33.755900 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751695 2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 19:00:33.755900 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751698 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 19:00:33.755900 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751701 2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 19:00:33.755900 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751704 2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 19:00:33.755900 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751706 2561 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 19:00:33.755900 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751709 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 19:00:33.755900 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751712 2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 19:00:33.755900 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751715 2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 19:00:33.755900 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751717 2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 19:00:33.755900 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751719 2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 19:00:33.755900 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751722 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 19:00:33.755900 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751724 2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 19:00:33.755900 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751727 2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 19:00:33.755900 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751730 2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 19:00:33.755900 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751733 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 19:00:33.755900 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751736 2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 19:00:33.755900 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751738 2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751741 2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751744 2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751746 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751749 2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751752 2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751755 2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751757 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751760 2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751762 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751764 2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751767 2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751769 2561 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751772 2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751775 2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751778 2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751780 2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751782 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751785 2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751787 2561 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 19:00:33.756376 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751790 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751793 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751795 2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751797 2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751800 2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751803 2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751807 2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751810 2561 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751813 2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751815 2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751817 2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751820 2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751822 2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751824 2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751827 2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751829 2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751832 2561 feature_gate.go:328] unrecognized feature gate: Example Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751835 2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751837 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751840 2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 19:00:33.756862 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751842 2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 19:00:33.757409 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751845 2561 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 19:00:33.757409 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751847 2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 19:00:33.757409 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751850 2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 19:00:33.757409 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751852 2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 19:00:33.757409 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751854 2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 19:00:33.757409 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751857 2561 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 19:00:33.757409 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751859 2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 19:00:33.757409 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751862 2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 19:00:33.757409 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751865 2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 19:00:33.757409 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751867 2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 19:00:33.757409 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751870 2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 19:00:33.757409 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751872 2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 19:00:33.757409 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751875 2561 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 19:00:33.757409 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751877 2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 19:00:33.757409 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751879 2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 19:00:33.757409 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751882 2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 19:00:33.757409 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:33.751884 2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 19:00:33.757822 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.751888 2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 19:00:33.757822 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.752033 2561 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 19:00:33.757822 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.754153 2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 19:00:33.757822 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.755784 2561 server.go:1019] "Starting client certificate rotation" Jul 10 19:00:33.757822 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.755878 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 19:00:33.757822 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.755910 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 19:00:33.758724 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.758708 2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 19:00:33.760001 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.759971 2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 19:00:33.771907 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.771879 2561 log.go:25] "Validated CRI v1 runtime API" Jul 10 19:00:33.774078 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.774058 2561 log.go:25] "Validated CRI v1 image API" Jul 10 19:00:33.775642 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.775626 2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 19:00:33.779475 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.779444 2561 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a03bdef9-4381-41f5-8c74-bbd4063353c6:/dev/nvme0n1p4 d3f8bfe1-fd9a-4266-a3bd-ff98fce83c36:/dev/nvme0n1p3] Jul 10 19:00:33.779539 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.779473 2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 19:00:33.784138 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.784121 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 19:00:33.785682 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.785577 2561 manager.go:216] Machine: {Timestamp:2026-07-10 19:00:33.784216405 +0000 UTC m=+0.385358552 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100731 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d34b539fb7abf1753f302799f3317 SystemUUID:ec2d34b5-39fb-7abf-1753-f302799f3317 BootID:3cacb2f2-3c2d-419f-9b64-3a00048a7fa7 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:42:80:2e:04:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:42:80:2e:04:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:cb:3c:d0:70:77 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 19:00:33.785682 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.785678 2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 19:00:33.785790 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.785758 2561 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 19:00:33.788973 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.788949 2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 19:00:33.789152 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.788975 2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-74.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 19:00:33.790632 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.790621 2561 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 19:00:33.790667 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.790635 2561 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 19:00:33.790667 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.790643 2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 19:00:33.791454 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.791443 2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 19:00:33.792303 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.792293 2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 19:00:33.795073 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.795059 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 10 19:00:33.795209 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.795199 2561 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 19:00:33.795252 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.795228 2561 kubelet.go:486] "Attempting to sync node with API server" Jul 10 19:00:33.795252 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.795238 2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 19:00:33.795319 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.795254 2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 19:00:33.795319 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.795262 2561 kubelet.go:398] "Adding apiserver pod source" Jul 10 19:00:33.795319 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.795272 2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 19:00:33.795753 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.795739 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 19:00:33.795791 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.795781 2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 19:00:33.798331 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.798305 2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 19:00:33.799835 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.799821 2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 19:00:33.799906 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.799851 2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 19:00:33.801036 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.801023 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 19:00:33.801090 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.801042 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 19:00:33.801090 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.801048 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 19:00:33.801090 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.801054 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 19:00:33.801090 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.801063 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 19:00:33.801090 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.801068 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 19:00:33.801090 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.801076 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 19:00:33.801090 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.801081 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 19:00:33.801090 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.801092 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 19:00:33.801326 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.801098 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 19:00:33.801326 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.801107 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 19:00:33.801326 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.801113 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 19:00:33.802017 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.802006 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 19:00:33.802017 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.802015 2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 19:00:33.807971 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.807928 2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-74.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 19:00:33.807971 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:33.807926 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 19:00:33.808103 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:33.808071 2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-74.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 19:00:33.808103 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.808098 2561 server.go:1268] "Started kubelet" Jul 10 19:00:33.808187 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.808144 2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 19:00:33.808263 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.808233 2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 19:00:33.808300 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.808289 2561 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 19:00:33.812609 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.812561 2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 19:00:33.813130 ip-10-0-130-74 systemd[1]: Started Kubernetes Kubelet. Jul 10 19:00:33.814839 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.813452 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j8zzk" Jul 10 19:00:33.815272 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.815258 2561 server.go:310] "Adding debug handlers to kubelet server" Jul 10 19:00:33.818717 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:33.818698 2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 19:00:33.818979 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.818964 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j8zzk" Jul 10 19:00:33.820239 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.820219 2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 19:00:33.820365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.820248 2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 19:00:33.820410 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:33.820379 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-74.ec2.internal\" not found" Jul 10 19:00:33.820901 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.820886 2561 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 19:00:33.820901 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.820903 2561 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 19:00:33.822417 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.822392 2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 19:00:33.823812 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:33.823785 2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-74.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 19:00:33.823812 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.823797 2561 reconstruct.go:97] "Volume reconstruction finished" Jul 10 19:00:33.823812 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.823819 2561 reconciler.go:29] "Reconciler: start to sync state" Jul 10 19:00:33.824085 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.824067 2561 factory.go:55] Registering systemd factory Jul 10 19:00:33.824138 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.824091 2561 factory.go:223] Registration of the systemd container factory successfully Jul 10 19:00:33.824416 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.824345 2561 factory.go:153] Registering CRI-O factory Jul 10 19:00:33.824416 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.824364 2561 factory.go:223] Registration of the crio container factory successfully Jul 10 19:00:33.824621 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.824431 2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 19:00:33.824621 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.824453 2561 factory.go:103] Registering Raw factory Jul 10 19:00:33.824621 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.824487 2561 manager.go:1196] Started watching for new ooms in manager Jul 10 19:00:33.825527 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.825514 2561 manager.go:349] Starting recovery of all containers Jul 10 19:00:33.835380 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.835258 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 19:00:33.836147 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.836131 2561 manager.go:354] Recovery completed Jul 10 19:00:33.840407 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.840393 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 19:00:33.842207 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.842189 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasSufficientMemory" Jul 10 19:00:33.842298 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.842221 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 19:00:33.842298 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.842234 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasSufficientPID" Jul 10 19:00:33.842848 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.842833 2561 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 19:00:33.842848 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.842845 2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 19:00:33.843047 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.842860 2561 state_mem.go:36] "Initialized new in-memory state store" Jul 10 19:00:33.843321 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.843307 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 19:00:33.844123 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.844107 2561 policy_none.go:49] "None policy: Start" Jul 10 19:00:33.844123 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.844124 2561 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 19:00:33.844253 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.844136 2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 19:00:33.844311 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.844296 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasSufficientMemory" Jul 10 19:00:33.844353 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.844330 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 19:00:33.844353 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.844343 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasSufficientPID" Jul 10 19:00:33.845426 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.845415 2561 policy_none.go:47] "Start" Jul 10 19:00:33.883524 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.883498 2561 manager.go:337] "Starting Device Plugin manager" Jul 10 19:00:33.883640 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:33.883540 2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 19:00:33.883640 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.883553 2561 server.go:87] "Starting device plugin registration server" Jul 10 19:00:33.883845 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.883832 2561 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 19:00:33.883900 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.883848 2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 19:00:33.883989 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.883971 2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 19:00:33.884099 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.884061 2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 19:00:33.884099 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.884070 2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 19:00:33.884618 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:33.884599 2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 19:00:33.884693 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:33.884644 2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-74.ec2.internal\" not found" Jul 10 19:00:33.955420 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.955375 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 19:00:33.956877 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.956782 2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 19:00:33.956877 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.956815 2561 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 19:00:33.956877 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.956838 2561 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 19:00:33.956877 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:33.956879 2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 19:00:33.959573 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.959555 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 19:00:33.985029 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.984915 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 19:00:33.986116 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.986096 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasSufficientMemory" Jul 10 19:00:33.986230 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.986131 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 19:00:33.986230 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.986142 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasSufficientPID" Jul 10 19:00:33.986230 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.986180 2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-74.ec2.internal" Jul 10 19:00:33.997969 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:33.997923 2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-74.ec2.internal" Jul 10 19:00:33.998120 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:33.997983 2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-74.ec2.internal\": node \"ip-10-0-130-74.ec2.internal\" not found" Jul 10 19:00:34.034159 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:34.034133 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-74.ec2.internal\" not found" Jul 10 19:00:34.057928 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.057867 2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-74.ec2.internal"] Jul 10 19:00:34.058083 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.057984 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 19:00:34.059219 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.059194 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasSufficientMemory" Jul 10 19:00:34.059312 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.059235 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 19:00:34.059312 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.059250 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasSufficientPID" Jul 10 19:00:34.060578 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.060565 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 19:00:34.060710 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.060694 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal" Jul 10 19:00:34.060749 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.060730 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 19:00:34.061593 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.061578 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasSufficientMemory" Jul 10 19:00:34.061675 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.061598 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasSufficientMemory" Jul 10 19:00:34.061675 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.061606 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 19:00:34.061675 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.061617 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 19:00:34.061675 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.061620 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasSufficientPID" Jul 10 19:00:34.061675 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.061627 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasSufficientPID" Jul 10 19:00:34.062881 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.062866 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-74.ec2.internal" Jul 10 19:00:34.062928 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.062896 2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 19:00:34.063837 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.063811 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasSufficientMemory" Jul 10 19:00:34.063902 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.063843 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 19:00:34.063902 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.063858 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeHasSufficientPID" Jul 10 19:00:34.086803 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:34.086778 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-74.ec2.internal\" not found" node="ip-10-0-130-74.ec2.internal" Jul 10 19:00:34.091581 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:34.091563 2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-74.ec2.internal\" not found" node="ip-10-0-130-74.ec2.internal" Jul 10 19:00:34.124989 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.124932 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c924a039fae58f41b47a54ccd1ac6a8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal\" (UID: \"8c924a039fae58f41b47a54ccd1ac6a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal" Jul 10 19:00:34.125104 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.125006 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c924a039fae58f41b47a54ccd1ac6a8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal\" (UID: \"8c924a039fae58f41b47a54ccd1ac6a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal" Jul 10 19:00:34.125104 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.125065 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7651e5e5e93604167127627171e7b52f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-74.ec2.internal\" (UID: \"7651e5e5e93604167127627171e7b52f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-74.ec2.internal" Jul 10 19:00:34.135259 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:34.135220 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-74.ec2.internal\" not found" Jul 10 19:00:34.225320 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.225280 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c924a039fae58f41b47a54ccd1ac6a8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal\" (UID: \"8c924a039fae58f41b47a54ccd1ac6a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal" Jul 10 19:00:34.225528 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.225350 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7651e5e5e93604167127627171e7b52f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-74.ec2.internal\" (UID: \"7651e5e5e93604167127627171e7b52f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-74.ec2.internal" Jul 10 19:00:34.225528 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.225373 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c924a039fae58f41b47a54ccd1ac6a8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal\" (UID: \"8c924a039fae58f41b47a54ccd1ac6a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal" Jul 10 19:00:34.225528 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.225278 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c924a039fae58f41b47a54ccd1ac6a8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal\" (UID: \"8c924a039fae58f41b47a54ccd1ac6a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal" Jul 10 19:00:34.225528 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.225414 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c924a039fae58f41b47a54ccd1ac6a8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal\" (UID: \"8c924a039fae58f41b47a54ccd1ac6a8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal" Jul 10 19:00:34.225528 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.225440 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7651e5e5e93604167127627171e7b52f-config\") pod \"kube-apiserver-proxy-ip-10-0-130-74.ec2.internal\" (UID: \"7651e5e5e93604167127627171e7b52f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-74.ec2.internal" Jul 10 19:00:34.235694 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:34.235626 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-74.ec2.internal\" not found" Jul 10 19:00:34.335976 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:34.335920 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-74.ec2.internal\" not found" Jul 10 19:00:34.393497 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.393465 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal" Jul 10 19:00:34.394614 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.394600 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-74.ec2.internal" Jul 10 19:00:34.436331 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:34.436284 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-74.ec2.internal\" not found" Jul 10 19:00:34.536785 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:34.536730 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-74.ec2.internal\" not found" Jul 10 19:00:34.637954 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:34.637897 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-74.ec2.internal\" not found" Jul 10 19:00:34.738429 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:34.738393 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-74.ec2.internal\" not found" Jul 10 19:00:34.754909 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.754878 2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 19:00:34.755119 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.755075 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 19:00:34.755162 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.755089 2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 19:00:34.823192 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.823118 2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 19:00:34.824108 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.824081 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 18:55:34 +0000 UTC" deadline="2027-12-30 12:07:03.889723786 +0000 UTC" Jul 10 19:00:34.824179 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:34.824110 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12905h6m29.065618225s" Jul 10 19:00:34.839081 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:34.839050 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-74.ec2.internal\" not found" Jul 10 19:00:34.940046 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:34.939992 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-74.ec2.internal\" not found" Jul 10 19:00:35.010211 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:35.010175 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7651e5e5e93604167127627171e7b52f.slice/crio-2693a1a425e3a2411721548596d98f641122954d296ca69779ed3f8b4191cecb WatchSource:0}: Error finding container 2693a1a425e3a2411721548596d98f641122954d296ca69779ed3f8b4191cecb: Status 404 returned error can't find the container with id 2693a1a425e3a2411721548596d98f641122954d296ca69779ed3f8b4191cecb Jul 10 19:00:35.011866 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:35.011848 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c924a039fae58f41b47a54ccd1ac6a8.slice/crio-9b56cb4dc1f3e5b13f6287c43a3fb272cf97d1cb1f21db2785b0663b37553542 WatchSource:0}: Error finding container 9b56cb4dc1f3e5b13f6287c43a3fb272cf97d1cb1f21db2785b0663b37553542: Status 404 returned error can't find the container with id 9b56cb4dc1f3e5b13f6287c43a3fb272cf97d1cb1f21db2785b0663b37553542 Jul 10 19:00:35.015259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.015240 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:00:35.040686 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:35.040656 2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-74.ec2.internal\" not found" Jul 10 19:00:35.101704 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.101636 2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 19:00:35.121037 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.121013 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal" Jul 10 19:00:35.133550 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.133527 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 19:00:35.134812 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.134798 2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-74.ec2.internal" Jul 10 19:00:35.144663 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.144641 2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 19:00:35.799294 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.799263 2561 apiserver.go:52] "Watching apiserver" Jul 10 19:00:35.807971 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.807928 2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 19:00:35.809291 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.809258 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-74.ec2.internal","openshift-cluster-node-tuning-operator/tuned-4c2qf","openshift-image-registry/node-ca-jb7v6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal","openshift-network-diagnostics/network-check-target-gwkcf","openshift-network-operator/iptables-alerter-xczgb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c","openshift-dns/node-resolver-gp2jn","openshift-multus/multus-additional-cni-plugins-5w2gc","openshift-multus/multus-d9nf8","openshift-multus/network-metrics-daemon-kpm26","openshift-ovn-kubernetes/ovnkube-node-knrq9","kube-system/global-pull-secret-syncer-j62sj","kube-system/konnectivity-agent-bxw74"] Jul 10 19:00:35.811410 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.810914 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.812263 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.812239 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:35.812370 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:35.812331 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j62sj" podUID="2509af5e-c021-4342-b2f3-7448be0ce972" Jul 10 19:00:35.813361 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.813339 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jb7v6" Jul 10 19:00:35.814290 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.814075 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 19:00:35.814290 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.814076 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 19:00:35.815120 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.814473 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bxw74" Jul 10 19:00:35.815120 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.814549 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 19:00:35.815120 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.814831 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zq5fg\"" Jul 10 19:00:35.815120 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.814940 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 19:00:35.821763 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.821728 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 19:00:35.821928 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.821816 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:35.821928 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:35.821897 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwkcf" podUID="3eb2cd17-af75-4a86-bf49-2ff369e4860d" Jul 10 19:00:35.822087 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.822030 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 19:00:35.822995 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.822975 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 19:00:35.823199 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.823141 2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 19:00:35.823199 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.823183 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.823199 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.823195 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.823363 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.823284 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xczgb" Jul 10 19:00:35.823363 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.823294 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gtkvf\"" Jul 10 19:00:35.823718 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.823689 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 19:00:35.823780 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.823726 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-q6jds\"" Jul 10 19:00:35.823998 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.823979 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.824662 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.824644 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 19:00:35.824739 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.824656 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gp2jn" Jul 10 19:00:35.826179 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.826158 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 19:00:35.826421 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.826402 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 19:00:35.826696 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.826674 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:35.826798 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:35.826770 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpm26" podUID="88ca28c8-725b-46f7-becc-ccd04207fba9" Jul 10 19:00:35.827724 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.827701 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.829473 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.828660 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 19:00:35.829473 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.828820 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 19:00:35.829473 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.828918 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 19:00:35.829473 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.828949 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 19:00:35.829473 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.828820 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 19:00:35.829473 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.829058 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 19:00:35.829473 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.829123 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rf7wg\"" Jul 10 19:00:35.829473 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.829171 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 19:00:35.829473 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.828867 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 19:00:35.829473 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.828861 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 19:00:35.829473 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.829256 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kf7ks\"" Jul 10 19:00:35.829473 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.829126 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4688h\"" Jul 10 19:00:35.829473 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.829171 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-skhgz\"" Jul 10 19:00:35.829473 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.829435 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p9chh\"" Jul 10 19:00:35.832887 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.831852 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 19:00:35.832887 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.831858 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 19:00:35.832887 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.832022 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nbsjf\"" Jul 10 19:00:35.832887 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.832218 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 19:00:35.832887 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.832223 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 19:00:35.832887 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.832455 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 19:00:35.833181 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.833027 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 19:00:35.835251 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.833890 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 19:00:35.835630 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.835584 2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 19:00:35.835907 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.835888 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-var-lib-kubelet\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.835973 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.835925 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6wjz\" (UniqueName: \"kubernetes.io/projected/d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855-kube-api-access-r6wjz\") pod \"iptables-alerter-xczgb\" (UID: \"d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855\") " pod="openshift-network-operator/iptables-alerter-xczgb" Jul 10 19:00:35.835973 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.835950 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-sys\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.836492 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.835977 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-tuned\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.836492 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836004 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/452ded98-132f-4e28-865a-d2f55032625c-hosts-file\") pod \"node-resolver-gp2jn\" (UID: \"452ded98-132f-4e28-865a-d2f55032625c\") " pod="openshift-dns/node-resolver-gp2jn" Jul 10 19:00:35.836492 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836051 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-os-release\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.836492 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836074 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-hostroot\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.836492 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836100 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1365527-36c3-4655-ac1a-eab0b02f92fd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.836492 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836166 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4766e5ae-0ae1-4fac-b87b-0b48178ee932-system-cni-dir\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.836492 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836203 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-multus-socket-dir-parent\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.836492 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836232 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-host-run-multus-certs\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.836492 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836260 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x699g\" (UniqueName: \"kubernetes.io/projected/ffe36bf2-7d7e-426f-a273-9acdbd6935e4-kube-api-access-x699g\") pod \"node-ca-jb7v6\" (UID: \"ffe36bf2-7d7e-426f-a273-9acdbd6935e4\") " pod="openshift-image-registry/node-ca-jb7v6" Jul 10 19:00:35.836492 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836288 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9ffcad3-0a20-40c1-9383-454676ea46aa-konnectivity-ca\") pod \"konnectivity-agent-bxw74\" (UID: \"f9ffcad3-0a20-40c1-9383-454676ea46aa\") " pod="kube-system/konnectivity-agent-bxw74" Jul 10 19:00:35.836492 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836313 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-sysctl-d\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.836492 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836335 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-lib-modules\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.836492 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836477 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-etc-kubernetes\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.838342 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836523 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9ffcad3-0a20-40c1-9383-454676ea46aa-agent-certs\") pod \"konnectivity-agent-bxw74\" (UID: \"f9ffcad3-0a20-40c1-9383-454676ea46aa\") " pod="kube-system/konnectivity-agent-bxw74" Jul 10 19:00:35.838342 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836559 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4766e5ae-0ae1-4fac-b87b-0b48178ee932-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.838342 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836587 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6j9z\" (UniqueName: \"kubernetes.io/projected/452ded98-132f-4e28-865a-d2f55032625c-kube-api-access-k6j9z\") pod \"node-resolver-gp2jn\" (UID: \"452ded98-132f-4e28-865a-d2f55032625c\") " pod="openshift-dns/node-resolver-gp2jn" Jul 10 19:00:35.838342 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836613 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffe36bf2-7d7e-426f-a273-9acdbd6935e4-serviceca\") pod \"node-ca-jb7v6\" (UID: \"ffe36bf2-7d7e-426f-a273-9acdbd6935e4\") " pod="openshift-image-registry/node-ca-jb7v6" Jul 10 19:00:35.838342 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836638 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4766e5ae-0ae1-4fac-b87b-0b48178ee932-cnibin\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.838342 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836662 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-host-var-lib-kubelet\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.838342 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836688 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2509af5e-c021-4342-b2f3-7448be0ce972-kubelet-config\") pod \"global-pull-secret-syncer-j62sj\" (UID: \"2509af5e-c021-4342-b2f3-7448be0ce972\") " pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:35.838342 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836713 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1365527-36c3-4655-ac1a-eab0b02f92fd-registration-dir\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.838342 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836814 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-host\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.838342 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836858 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfecda89-a83d-44a7-88a9-e053a3a6e311-tmp\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.838342 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836889 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4766e5ae-0ae1-4fac-b87b-0b48178ee932-os-release\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.838342 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836909 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4766e5ae-0ae1-4fac-b87b-0b48178ee932-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.838342 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836964 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-run\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.838342 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.836994 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1365527-36c3-4655-ac1a-eab0b02f92fd-etc-selinux\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.838342 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837021 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvmgs\" (UniqueName: \"kubernetes.io/projected/4766e5ae-0ae1-4fac-b87b-0b48178ee932-kube-api-access-xvmgs\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.838342 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837045 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-systemd\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.840851 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837069 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz6g6\" (UniqueName: \"kubernetes.io/projected/cfecda89-a83d-44a7-88a9-e053a3a6e311-kube-api-access-fz6g6\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.840851 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837093 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-multus-conf-dir\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.840851 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837119 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/54a84657-5511-4d5f-bc6b-6602b5b2c654-multus-daemon-config\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.840851 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837199 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvmlp\" (UniqueName: \"kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp\") pod \"network-check-target-gwkcf\" (UID: \"3eb2cd17-af75-4a86-bf49-2ff369e4860d\") " pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:35.840851 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837228 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-host-var-lib-cni-multus\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.840851 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837261 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2509af5e-c021-4342-b2f3-7448be0ce972-dbus\") pod \"global-pull-secret-syncer-j62sj\" (UID: \"2509af5e-c021-4342-b2f3-7448be0ce972\") " pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:35.840851 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837285 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-sysctl-conf\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.840851 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837332 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-host-run-netns\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.840851 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837355 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-host-var-lib-cni-bin\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.840851 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837382 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret\") pod \"global-pull-secret-syncer-j62sj\" (UID: \"2509af5e-c021-4342-b2f3-7448be0ce972\") " pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:35.840851 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837416 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1365527-36c3-4655-ac1a-eab0b02f92fd-socket-dir\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.840851 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837456 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855-host-slash\") pod \"iptables-alerter-xczgb\" (UID: \"d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855\") " pod="openshift-network-operator/iptables-alerter-xczgb" Jul 10 19:00:35.840851 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837491 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4766e5ae-0ae1-4fac-b87b-0b48178ee932-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.840851 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837517 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-kubernetes\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.840851 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837542 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-system-cni-dir\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.840851 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837568 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fghqr\" (UniqueName: \"kubernetes.io/projected/f1365527-36c3-4655-ac1a-eab0b02f92fd-kube-api-access-fghqr\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.841665 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837603 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4766e5ae-0ae1-4fac-b87b-0b48178ee932-cni-binary-copy\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.841665 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837628 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54a84657-5511-4d5f-bc6b-6602b5b2c654-cni-binary-copy\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.841665 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837654 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-host-run-k8s-cni-cncf-io\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.841665 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837688 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gcth\" (UniqueName: \"kubernetes.io/projected/54a84657-5511-4d5f-bc6b-6602b5b2c654-kube-api-access-9gcth\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.841665 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837721 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855-iptables-alerter-script\") pod \"iptables-alerter-xczgb\" (UID: \"d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855\") " pod="openshift-network-operator/iptables-alerter-xczgb" Jul 10 19:00:35.841665 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837745 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-sysconfig\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.841665 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837788 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452ded98-132f-4e28-865a-d2f55032625c-tmp-dir\") pod \"node-resolver-gp2jn\" (UID: \"452ded98-132f-4e28-865a-d2f55032625c\") " pod="openshift-dns/node-resolver-gp2jn" Jul 10 19:00:35.841665 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837811 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-multus-cni-dir\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.841665 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837850 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffe36bf2-7d7e-426f-a273-9acdbd6935e4-host\") pod \"node-ca-jb7v6\" (UID: \"ffe36bf2-7d7e-426f-a273-9acdbd6935e4\") " pod="openshift-image-registry/node-ca-jb7v6" Jul 10 19:00:35.841665 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837874 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1365527-36c3-4655-ac1a-eab0b02f92fd-sys-fs\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.841665 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837897 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-modprobe-d\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.841665 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837925 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-cnibin\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.841665 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.837948 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1365527-36c3-4655-ac1a-eab0b02f92fd-device-dir\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.861879 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.861855 2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-89cxt" Jul 10 19:00:35.868779 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.868756 2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-89cxt" Jul 10 19:00:35.924048 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.924004 2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 19:00:35.938701 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.938666 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6j9z\" (UniqueName: \"kubernetes.io/projected/452ded98-132f-4e28-865a-d2f55032625c-kube-api-access-k6j9z\") pod \"node-resolver-gp2jn\" (UID: \"452ded98-132f-4e28-865a-d2f55032625c\") " pod="openshift-dns/node-resolver-gp2jn" Jul 10 19:00:35.938911 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.938714 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-run-netns\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.938911 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.938750 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-run-ovn-kubernetes\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.938911 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.938788 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffe36bf2-7d7e-426f-a273-9acdbd6935e4-serviceca\") pod \"node-ca-jb7v6\" (UID: \"ffe36bf2-7d7e-426f-a273-9acdbd6935e4\") " pod="openshift-image-registry/node-ca-jb7v6" Jul 10 19:00:35.938911 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.938813 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4766e5ae-0ae1-4fac-b87b-0b48178ee932-cnibin\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.938911 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.938836 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-host-var-lib-kubelet\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.938911 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.938868 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2509af5e-c021-4342-b2f3-7448be0ce972-kubelet-config\") pod \"global-pull-secret-syncer-j62sj\" (UID: \"2509af5e-c021-4342-b2f3-7448be0ce972\") " pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:35.938911 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.938894 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1365527-36c3-4655-ac1a-eab0b02f92fd-registration-dir\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.939316 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.938927 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-kubelet\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.939316 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.938950 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-run-ovn\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.939316 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939032 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-host\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.939316 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939058 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfecda89-a83d-44a7-88a9-e053a3a6e311-tmp\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.939316 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939085 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-var-lib-openvswitch\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.939316 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939112 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-cni-netd\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.939316 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939143 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-ovn-node-metrics-cert\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.939316 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939174 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4766e5ae-0ae1-4fac-b87b-0b48178ee932-os-release\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.939316 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939204 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4766e5ae-0ae1-4fac-b87b-0b48178ee932-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.939316 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939248 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-run\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.939316 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939281 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1365527-36c3-4655-ac1a-eab0b02f92fd-etc-selinux\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.939316 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939307 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvmgs\" (UniqueName: \"kubernetes.io/projected/4766e5ae-0ae1-4fac-b87b-0b48178ee932-kube-api-access-xvmgs\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.939951 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939331 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-systemd\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.939951 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939356 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz6g6\" (UniqueName: \"kubernetes.io/projected/cfecda89-a83d-44a7-88a9-e053a3a6e311-kube-api-access-fz6g6\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.939951 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939408 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-multus-conf-dir\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.939951 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939464 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/54a84657-5511-4d5f-bc6b-6602b5b2c654-multus-daemon-config\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.939951 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939502 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmlp\" (UniqueName: \"kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp\") pod \"network-check-target-gwkcf\" (UID: \"3eb2cd17-af75-4a86-bf49-2ff369e4860d\") " pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:35.939951 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939528 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-env-overrides\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.939951 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939559 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-host-var-lib-cni-multus\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.939951 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939597 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2509af5e-c021-4342-b2f3-7448be0ce972-dbus\") pod \"global-pull-secret-syncer-j62sj\" (UID: \"2509af5e-c021-4342-b2f3-7448be0ce972\") " pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:35.939951 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939623 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-cni-bin\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.939951 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939662 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-ovnkube-config\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.939951 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939705 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-sysctl-conf\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.939951 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939732 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-host-run-netns\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.939951 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939776 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-host-var-lib-cni-bin\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.939951 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939867 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret\") pod \"global-pull-secret-syncer-j62sj\" (UID: \"2509af5e-c021-4342-b2f3-7448be0ce972\") " pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:35.939951 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939906 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1365527-36c3-4655-ac1a-eab0b02f92fd-socket-dir\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.939951 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939932 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffe36bf2-7d7e-426f-a273-9acdbd6935e4-serviceca\") pod \"node-ca-jb7v6\" (UID: \"ffe36bf2-7d7e-426f-a273-9acdbd6935e4\") " pod="openshift-image-registry/node-ca-jb7v6" Jul 10 19:00:35.940815 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939961 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855-host-slash\") pod \"iptables-alerter-xczgb\" (UID: \"d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855\") " pod="openshift-network-operator/iptables-alerter-xczgb" Jul 10 19:00:35.940815 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939994 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-run-openvswitch\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.940815 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.940037 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-ovnkube-script-lib\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.940815 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.940065 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4766e5ae-0ae1-4fac-b87b-0b48178ee932-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.940815 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.940116 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-kubernetes\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.940815 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.940144 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-system-cni-dir\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.940815 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.940145 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-systemd\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.940815 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.939055 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4766e5ae-0ae1-4fac-b87b-0b48178ee932-cnibin\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.940815 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.940690 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-multus-conf-dir\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.941329 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.941099 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-host-var-lib-cni-bin\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.941329 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.941133 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fghqr\" (UniqueName: \"kubernetes.io/projected/f1365527-36c3-4655-ac1a-eab0b02f92fd-kube-api-access-fghqr\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.941329 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.941178 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-etc-openvswitch\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.941329 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.941197 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4766e5ae-0ae1-4fac-b87b-0b48178ee932-cni-binary-copy\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.941329 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.941270 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54a84657-5511-4d5f-bc6b-6602b5b2c654-cni-binary-copy\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.941329 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.941289 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-host-run-k8s-cni-cncf-io\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.941329 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.941297 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-host-run-netns\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.941329 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.941305 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gcth\" (UniqueName: \"kubernetes.io/projected/54a84657-5511-4d5f-bc6b-6602b5b2c654-kube-api-access-9gcth\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.941329 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.941333 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855-iptables-alerter-script\") pod \"iptables-alerter-xczgb\" (UID: \"d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855\") " pod="openshift-network-operator/iptables-alerter-xczgb" Jul 10 19:00:35.941793 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.941350 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rwfm\" (UniqueName: \"kubernetes.io/projected/88ca28c8-725b-46f7-becc-ccd04207fba9-kube-api-access-6rwfm\") pod \"network-metrics-daemon-kpm26\" (UID: \"88ca28c8-725b-46f7-becc-ccd04207fba9\") " pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:35.941793 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.941367 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-sysconfig\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.941793 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.941384 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452ded98-132f-4e28-865a-d2f55032625c-tmp-dir\") pod \"node-resolver-gp2jn\" (UID: \"452ded98-132f-4e28-865a-d2f55032625c\") " pod="openshift-dns/node-resolver-gp2jn" Jul 10 19:00:35.941793 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.941425 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-multus-cni-dir\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.942739 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.941442 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/54a84657-5511-4d5f-bc6b-6602b5b2c654-multus-daemon-config\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.942739 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.942364 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffe36bf2-7d7e-426f-a273-9acdbd6935e4-host\") pod \"node-ca-jb7v6\" (UID: \"ffe36bf2-7d7e-426f-a273-9acdbd6935e4\") " pod="openshift-image-registry/node-ca-jb7v6" Jul 10 19:00:35.942739 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.942442 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f1365527-36c3-4655-ac1a-eab0b02f92fd-etc-selinux\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.942739 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.942454 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1365527-36c3-4655-ac1a-eab0b02f92fd-sys-fs\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.942739 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.942539 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f1365527-36c3-4655-ac1a-eab0b02f92fd-sys-fs\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.942739 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.942590 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-host-run-k8s-cni-cncf-io\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.942739 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.942696 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4766e5ae-0ae1-4fac-b87b-0b48178ee932-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.943558 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.942737 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2509af5e-c021-4342-b2f3-7448be0ce972-kubelet-config\") pod \"global-pull-secret-syncer-j62sj\" (UID: \"2509af5e-c021-4342-b2f3-7448be0ce972\") " pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:35.943558 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.942738 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-host\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.943558 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.942973 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f1365527-36c3-4655-ac1a-eab0b02f92fd-registration-dir\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.943558 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.943031 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54a84657-5511-4d5f-bc6b-6602b5b2c654-cni-binary-copy\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.943558 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.943135 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-systemd-units\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.943558 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.943335 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-sysconfig\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.943558 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.943343 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.943558 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.943360 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4766e5ae-0ae1-4fac-b87b-0b48178ee932-os-release\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.943558 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.943405 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-modprobe-d\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.943558 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.943439 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-multus-cni-dir\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.944106 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:35.943607 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:35.944106 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.943619 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/452ded98-132f-4e28-865a-d2f55032625c-tmp-dir\") pod \"node-resolver-gp2jn\" (UID: \"452ded98-132f-4e28-865a-d2f55032625c\") " pod="openshift-dns/node-resolver-gp2jn" Jul 10 19:00:35.944106 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.943650 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-modprobe-d\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.944106 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.943638 2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 19:00:35.944106 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.943796 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-host-var-lib-cni-multus\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.944106 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:35.943834 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret podName:2509af5e-c021-4342-b2f3-7448be0ce972 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:36.443792962 +0000 UTC m=+3.044935094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret") pod "global-pull-secret-syncer-j62sj" (UID: "2509af5e-c021-4342-b2f3-7448be0ce972") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:35.944106 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.943852 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-system-cni-dir\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.944459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.944144 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855-host-slash\") pod \"iptables-alerter-xczgb\" (UID: \"d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855\") " pod="openshift-network-operator/iptables-alerter-xczgb" Jul 10 19:00:35.944459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.944322 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f1365527-36c3-4655-ac1a-eab0b02f92fd-socket-dir\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.944459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.944377 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-sysctl-conf\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.944600 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.944450 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-host-var-lib-kubelet\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.944600 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.944568 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-run\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.944695 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.944630 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2509af5e-c021-4342-b2f3-7448be0ce972-dbus\") pod \"global-pull-secret-syncer-j62sj\" (UID: \"2509af5e-c021-4342-b2f3-7448be0ce972\") " pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:35.947210 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.944877 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-cnibin\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.947210 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945253 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1365527-36c3-4655-ac1a-eab0b02f92fd-device-dir\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.947210 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945303 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-var-lib-kubelet\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.947210 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945339 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4766e5ae-0ae1-4fac-b87b-0b48178ee932-cni-binary-copy\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.947210 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945352 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6wjz\" (UniqueName: \"kubernetes.io/projected/d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855-kube-api-access-r6wjz\") pod \"iptables-alerter-xczgb\" (UID: \"d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855\") " pod="openshift-network-operator/iptables-alerter-xczgb" Jul 10 19:00:35.947210 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945386 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-slash\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.947210 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945466 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f1365527-36c3-4655-ac1a-eab0b02f92fd-device-dir\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.947210 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945483 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-var-lib-kubelet\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.947210 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945459 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4766e5ae-0ae1-4fac-b87b-0b48178ee932-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.947210 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945517 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-sys\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.947210 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945570 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffe36bf2-7d7e-426f-a273-9acdbd6935e4-host\") pod \"node-ca-jb7v6\" (UID: \"ffe36bf2-7d7e-426f-a273-9acdbd6935e4\") " pod="openshift-image-registry/node-ca-jb7v6" Jul 10 19:00:35.947210 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945575 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-tuned\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.947210 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945609 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/452ded98-132f-4e28-865a-d2f55032625c-hosts-file\") pod \"node-resolver-gp2jn\" (UID: \"452ded98-132f-4e28-865a-d2f55032625c\") " pod="openshift-dns/node-resolver-gp2jn" Jul 10 19:00:35.947210 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945634 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-kubernetes\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.947210 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945639 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-os-release\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.947210 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945676 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-hostroot\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.947210 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945713 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1365527-36c3-4655-ac1a-eab0b02f92fd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.948259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945751 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-run-systemd\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.948259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945785 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-node-log\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.948259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945818 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-log-socket\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.948259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945855 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwxrk\" (UniqueName: \"kubernetes.io/projected/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-kube-api-access-bwxrk\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:35.948259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945895 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs\") pod \"network-metrics-daemon-kpm26\" (UID: \"88ca28c8-725b-46f7-becc-ccd04207fba9\") " pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:35.948259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945955 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4766e5ae-0ae1-4fac-b87b-0b48178ee932-system-cni-dir\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.948259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.945989 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-multus-socket-dir-parent\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.948259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946019 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f1365527-36c3-4655-ac1a-eab0b02f92fd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.948259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946034 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-host-run-multus-certs\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.948259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946086 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/452ded98-132f-4e28-865a-d2f55032625c-hosts-file\") pod \"node-resolver-gp2jn\" (UID: \"452ded98-132f-4e28-865a-d2f55032625c\") " pod="openshift-dns/node-resolver-gp2jn" Jul 10 19:00:35.948259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946101 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x699g\" (UniqueName: \"kubernetes.io/projected/ffe36bf2-7d7e-426f-a273-9acdbd6935e4-kube-api-access-x699g\") pod \"node-ca-jb7v6\" (UID: \"ffe36bf2-7d7e-426f-a273-9acdbd6935e4\") " pod="openshift-image-registry/node-ca-jb7v6" Jul 10 19:00:35.948259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946136 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9ffcad3-0a20-40c1-9383-454676ea46aa-konnectivity-ca\") pod \"konnectivity-agent-bxw74\" (UID: \"f9ffcad3-0a20-40c1-9383-454676ea46aa\") " pod="kube-system/konnectivity-agent-bxw74" Jul 10 19:00:35.948259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946156 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-os-release\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.948259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946175 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-sysctl-d\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.948259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946209 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-hostroot\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.948259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946210 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-lib-modules\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.948259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946255 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-etc-kubernetes\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.949447 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946268 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-cnibin\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.949447 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946288 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9ffcad3-0a20-40c1-9383-454676ea46aa-agent-certs\") pod \"konnectivity-agent-bxw74\" (UID: \"f9ffcad3-0a20-40c1-9383-454676ea46aa\") " pod="kube-system/konnectivity-agent-bxw74" Jul 10 19:00:35.949447 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946321 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-sys\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.949447 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946325 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4766e5ae-0ae1-4fac-b87b-0b48178ee932-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.949447 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946388 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-multus-socket-dir-parent\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.949447 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946463 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-host-run-multus-certs\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.949447 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946538 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4766e5ae-0ae1-4fac-b87b-0b48178ee932-system-cni-dir\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.949447 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946683 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-sysctl-d\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.949447 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946767 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54a84657-5511-4d5f-bc6b-6602b5b2c654-etc-kubernetes\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.949447 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.946993 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cfecda89-a83d-44a7-88a9-e053a3a6e311-lib-modules\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.949447 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.947118 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f9ffcad3-0a20-40c1-9383-454676ea46aa-konnectivity-ca\") pod \"konnectivity-agent-bxw74\" (UID: \"f9ffcad3-0a20-40c1-9383-454676ea46aa\") " pod="kube-system/konnectivity-agent-bxw74" Jul 10 19:00:35.949447 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.948340 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855-iptables-alerter-script\") pod \"iptables-alerter-xczgb\" (UID: \"d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855\") " pod="openshift-network-operator/iptables-alerter-xczgb" Jul 10 19:00:35.949447 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.948370 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4766e5ae-0ae1-4fac-b87b-0b48178ee932-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.950666 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.950640 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cfecda89-a83d-44a7-88a9-e053a3a6e311-etc-tuned\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.951625 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.951562 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cfecda89-a83d-44a7-88a9-e053a3a6e311-tmp\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.952418 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.952388 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f9ffcad3-0a20-40c1-9383-454676ea46aa-agent-certs\") pod \"konnectivity-agent-bxw74\" (UID: \"f9ffcad3-0a20-40c1-9383-454676ea46aa\") " pod="kube-system/konnectivity-agent-bxw74" Jul 10 19:00:35.956058 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:35.954714 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:00:35.956058 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:35.954740 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:00:35.956058 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:35.954755 2561 projected.go:196] Error preparing data for projected volume kube-api-access-qvmlp for pod openshift-network-diagnostics/network-check-target-gwkcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:35.956058 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:35.954820 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp podName:3eb2cd17-af75-4a86-bf49-2ff369e4860d nodeName:}" failed. No retries permitted until 2026-07-10 19:00:36.454799352 +0000 UTC m=+3.055941489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qvmlp" (UniqueName: "kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp") pod "network-check-target-gwkcf" (UID: "3eb2cd17-af75-4a86-bf49-2ff369e4860d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:35.960195 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.959229 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz6g6\" (UniqueName: \"kubernetes.io/projected/cfecda89-a83d-44a7-88a9-e053a3a6e311-kube-api-access-fz6g6\") pod \"tuned-4c2qf\" (UID: \"cfecda89-a83d-44a7-88a9-e053a3a6e311\") " pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:35.961890 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.961841 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gcth\" (UniqueName: \"kubernetes.io/projected/54a84657-5511-4d5f-bc6b-6602b5b2c654-kube-api-access-9gcth\") pod \"multus-d9nf8\" (UID: \"54a84657-5511-4d5f-bc6b-6602b5b2c654\") " pod="openshift-multus/multus-d9nf8" Jul 10 19:00:35.964154 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.964129 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6j9z\" (UniqueName: \"kubernetes.io/projected/452ded98-132f-4e28-865a-d2f55032625c-kube-api-access-k6j9z\") pod \"node-resolver-gp2jn\" (UID: \"452ded98-132f-4e28-865a-d2f55032625c\") " pod="openshift-dns/node-resolver-gp2jn" Jul 10 19:00:35.966772 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.966672 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvmgs\" (UniqueName: \"kubernetes.io/projected/4766e5ae-0ae1-4fac-b87b-0b48178ee932-kube-api-access-xvmgs\") pod \"multus-additional-cni-plugins-5w2gc\" (UID: \"4766e5ae-0ae1-4fac-b87b-0b48178ee932\") " pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:35.966772 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.966679 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-74.ec2.internal" event={"ID":"7651e5e5e93604167127627171e7b52f","Type":"ContainerStarted","Data":"2693a1a425e3a2411721548596d98f641122954d296ca69779ed3f8b4191cecb"} Jul 10 19:00:35.968042 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.967117 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6wjz\" (UniqueName: \"kubernetes.io/projected/d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855-kube-api-access-r6wjz\") pod \"iptables-alerter-xczgb\" (UID: \"d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855\") " pod="openshift-network-operator/iptables-alerter-xczgb" Jul 10 19:00:35.968042 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.967153 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x699g\" (UniqueName: \"kubernetes.io/projected/ffe36bf2-7d7e-426f-a273-9acdbd6935e4-kube-api-access-x699g\") pod \"node-ca-jb7v6\" (UID: \"ffe36bf2-7d7e-426f-a273-9acdbd6935e4\") " pod="openshift-image-registry/node-ca-jb7v6" Jul 10 19:00:35.969048 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.968893 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fghqr\" (UniqueName: \"kubernetes.io/projected/f1365527-36c3-4655-ac1a-eab0b02f92fd-kube-api-access-fghqr\") pod \"aws-ebs-csi-driver-node-7fb4c\" (UID: \"f1365527-36c3-4655-ac1a-eab0b02f92fd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:35.969430 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:35.969404 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal" event={"ID":"8c924a039fae58f41b47a54ccd1ac6a8","Type":"ContainerStarted","Data":"9b56cb4dc1f3e5b13f6287c43a3fb272cf97d1cb1f21db2785b0663b37553542"} Jul 10 19:00:36.047101 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047058 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047312 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047112 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-slash\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047312 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047148 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-run-systemd\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047312 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047173 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-node-log\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047312 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047191 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047312 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047255 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-slash\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047312 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047263 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-log-socket\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047312 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047202 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-log-socket\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047312 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047303 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-run-systemd\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047312 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047306 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxrk\" (UniqueName: \"kubernetes.io/projected/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-kube-api-access-bwxrk\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047779 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047334 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs\") pod \"network-metrics-daemon-kpm26\" (UID: \"88ca28c8-725b-46f7-becc-ccd04207fba9\") " pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:36.047779 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047344 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-node-log\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047779 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047372 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-run-netns\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047779 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047399 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-run-ovn-kubernetes\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047779 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047431 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-kubelet\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047779 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047471 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-run-ovn\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047779 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047497 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-var-lib-openvswitch\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047779 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047524 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-cni-netd\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047779 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047548 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-ovn-node-metrics-cert\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047779 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047598 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-env-overrides\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047779 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047625 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-cni-bin\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047779 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047646 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-kubelet\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047779 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047715 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-run-ovn\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047779 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:36.047722 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:36.047779 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047761 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-var-lib-openvswitch\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.047779 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:36.047785 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs podName:88ca28c8-725b-46f7-becc-ccd04207fba9 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:36.547770715 +0000 UTC m=+3.148912830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs") pod "network-metrics-daemon-kpm26" (UID: "88ca28c8-725b-46f7-becc-ccd04207fba9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:36.048629 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047807 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-cni-netd\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.048629 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047812 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-run-netns\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.048629 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047841 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-run-ovn-kubernetes\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.048629 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047873 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-ovnkube-config\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.048629 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047906 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-run-openvswitch\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.048629 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047930 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-ovnkube-script-lib\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.048629 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047953 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-etc-openvswitch\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.048629 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047974 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rwfm\" (UniqueName: \"kubernetes.io/projected/88ca28c8-725b-46f7-becc-ccd04207fba9-kube-api-access-6rwfm\") pod \"network-metrics-daemon-kpm26\" (UID: \"88ca28c8-725b-46f7-becc-ccd04207fba9\") " pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:36.048629 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.047996 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-systemd-units\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.048629 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.048063 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-systemd-units\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.048629 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.048266 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-etc-openvswitch\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.048629 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.048318 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-run-openvswitch\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.049180 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.048812 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-ovnkube-config\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.049180 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.048825 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-ovnkube-script-lib\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.049180 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.049081 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-host-cni-bin\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.049421 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.049347 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-env-overrides\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.052435 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.052407 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-ovn-node-metrics-cert\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.059570 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.059532 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwxrk\" (UniqueName: \"kubernetes.io/projected/7c3b76b0-8cb8-489c-ae2e-a8e78982c24b-kube-api-access-bwxrk\") pod \"ovnkube-node-knrq9\" (UID: \"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b\") " pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.059700 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.059603 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rwfm\" (UniqueName: \"kubernetes.io/projected/88ca28c8-725b-46f7-becc-ccd04207fba9-kube-api-access-6rwfm\") pod \"network-metrics-daemon-kpm26\" (UID: \"88ca28c8-725b-46f7-becc-ccd04207fba9\") " pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:36.076615 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.076581 2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 19:00:36.135642 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.135604 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-d9nf8" Jul 10 19:00:36.146768 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.146737 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jb7v6" Jul 10 19:00:36.148689 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:36.148659 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a84657_5511_4d5f_bc6b_6602b5b2c654.slice/crio-0ed111494830e4998acc6a0f5ce9ef51fa67e01b89c810c0107382a1f58a0e7f WatchSource:0}: Error finding container 0ed111494830e4998acc6a0f5ce9ef51fa67e01b89c810c0107382a1f58a0e7f: Status 404 returned error can't find the container with id 0ed111494830e4998acc6a0f5ce9ef51fa67e01b89c810c0107382a1f58a0e7f Jul 10 19:00:36.158055 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:36.157950 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe36bf2_7d7e_426f_a273_9acdbd6935e4.slice/crio-f3c901fc5b51a816fe667cbf5eedce92435b16849f97bec2d3f8d7809241ec89 WatchSource:0}: Error finding container f3c901fc5b51a816fe667cbf5eedce92435b16849f97bec2d3f8d7809241ec89: Status 404 returned error can't find the container with id f3c901fc5b51a816fe667cbf5eedce92435b16849f97bec2d3f8d7809241ec89 Jul 10 19:00:36.159562 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.159230 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bxw74" Jul 10 19:00:36.165416 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.165385 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" Jul 10 19:00:36.176204 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:36.176173 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1365527_36c3_4655_ac1a_eab0b02f92fd.slice/crio-5f105c70d549eaaff2e641e557dbade9c0b83960770896c4b9588999ad1426ec WatchSource:0}: Error finding container 5f105c70d549eaaff2e641e557dbade9c0b83960770896c4b9588999ad1426ec: Status 404 returned error can't find the container with id 5f105c70d549eaaff2e641e557dbade9c0b83960770896c4b9588999ad1426ec Jul 10 19:00:36.176470 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.176447 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xczgb" Jul 10 19:00:36.183281 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.182999 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5w2gc" Jul 10 19:00:36.189583 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:36.189561 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49dce93_b7fa_4c9a_b97f_dc3a9ca6b855.slice/crio-5baf1dac02a3cc348f78d94f0e81ef8383ef4d6e1a48b4609e2eb0e5ab4987c2 WatchSource:0}: Error finding container 5baf1dac02a3cc348f78d94f0e81ef8383ef4d6e1a48b4609e2eb0e5ab4987c2: Status 404 returned error can't find the container with id 5baf1dac02a3cc348f78d94f0e81ef8383ef4d6e1a48b4609e2eb0e5ab4987c2 Jul 10 19:00:36.191728 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.191636 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" Jul 10 19:00:36.194722 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:36.194691 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4766e5ae_0ae1_4fac_b87b_0b48178ee932.slice/crio-8c325f698db4c9641960737eee2b693dc6d58abb70a5230034b810d5b5b152ce WatchSource:0}: Error finding container 8c325f698db4c9641960737eee2b693dc6d58abb70a5230034b810d5b5b152ce: Status 404 returned error can't find the container with id 8c325f698db4c9641960737eee2b693dc6d58abb70a5230034b810d5b5b152ce Jul 10 19:00:36.204539 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:36.204513 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfecda89_a83d_44a7_88a9_e053a3a6e311.slice/crio-cb4756e1aa59df1136f98886e4e3fc594f46468ed3d1629dc23ab4ce92943a2c WatchSource:0}: Error finding container cb4756e1aa59df1136f98886e4e3fc594f46468ed3d1629dc23ab4ce92943a2c: Status 404 returned error can't find the container with id cb4756e1aa59df1136f98886e4e3fc594f46468ed3d1629dc23ab4ce92943a2c Jul 10 19:00:36.207594 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.207580 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gp2jn" Jul 10 19:00:36.214558 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.214347 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:00:36.223095 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:36.219468 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452ded98_132f_4e28_865a_d2f55032625c.slice/crio-3510e53ada1bf73e15481d4c211e9761f0f9d1148df15a1f75fdeea4ef519aa2 WatchSource:0}: Error finding container 3510e53ada1bf73e15481d4c211e9761f0f9d1148df15a1f75fdeea4ef519aa2: Status 404 returned error can't find the container with id 3510e53ada1bf73e15481d4c211e9761f0f9d1148df15a1f75fdeea4ef519aa2 Jul 10 19:00:36.230136 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:00:36.230092 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3b76b0_8cb8_489c_ae2e_a8e78982c24b.slice/crio-ed97b67a959db7f27cd6cf247dd0112105b3e4c5babc6466904eb6f726bb1ba5 WatchSource:0}: Error finding container ed97b67a959db7f27cd6cf247dd0112105b3e4c5babc6466904eb6f726bb1ba5: Status 404 returned error can't find the container with id ed97b67a959db7f27cd6cf247dd0112105b3e4c5babc6466904eb6f726bb1ba5 Jul 10 19:00:36.294624 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.294598 2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 19:00:36.451347 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.451305 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret\") pod \"global-pull-secret-syncer-j62sj\" (UID: \"2509af5e-c021-4342-b2f3-7448be0ce972\") " pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:36.451544 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:36.451487 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:36.451603 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:36.451581 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret podName:2509af5e-c021-4342-b2f3-7448be0ce972 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:37.451558435 +0000 UTC m=+4.052700574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret") pod "global-pull-secret-syncer-j62sj" (UID: "2509af5e-c021-4342-b2f3-7448be0ce972") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:36.552753 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.552710 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs\") pod \"network-metrics-daemon-kpm26\" (UID: \"88ca28c8-725b-46f7-becc-ccd04207fba9\") " pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:36.552933 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.552779 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmlp\" (UniqueName: \"kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp\") pod \"network-check-target-gwkcf\" (UID: \"3eb2cd17-af75-4a86-bf49-2ff369e4860d\") " pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:36.552933 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:36.552847 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:36.552933 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:36.552901 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:00:36.552933 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:36.552918 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:00:36.552933 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:36.552924 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs podName:88ca28c8-725b-46f7-becc-ccd04207fba9 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:37.55290331 +0000 UTC m=+4.154045427 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs") pod "network-metrics-daemon-kpm26" (UID: "88ca28c8-725b-46f7-becc-ccd04207fba9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:36.552933 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:36.552932 2561 projected.go:196] Error preparing data for projected volume kube-api-access-qvmlp for pod openshift-network-diagnostics/network-check-target-gwkcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:36.553353 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:36.552988 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp podName:3eb2cd17-af75-4a86-bf49-2ff369e4860d nodeName:}" failed. No retries permitted until 2026-07-10 19:00:37.552959652 +0000 UTC m=+4.154101782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvmlp" (UniqueName: "kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp") pod "network-check-target-gwkcf" (UID: "3eb2cd17-af75-4a86-bf49-2ff369e4860d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:36.871754 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.871691 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:55:36 +0000 UTC" deadline="2028-02-26 08:58:06.686327721 +0000 UTC" Jul 10 19:00:36.871754 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.871744 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14293h57m29.814588021s" Jul 10 19:00:36.973855 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.973813 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" event={"ID":"cfecda89-a83d-44a7-88a9-e053a3a6e311","Type":"ContainerStarted","Data":"cb4756e1aa59df1136f98886e4e3fc594f46468ed3d1629dc23ab4ce92943a2c"} Jul 10 19:00:36.975400 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.975330 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5w2gc" event={"ID":"4766e5ae-0ae1-4fac-b87b-0b48178ee932","Type":"ContainerStarted","Data":"8c325f698db4c9641960737eee2b693dc6d58abb70a5230034b810d5b5b152ce"} Jul 10 19:00:36.976686 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.976651 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" event={"ID":"f1365527-36c3-4655-ac1a-eab0b02f92fd","Type":"ContainerStarted","Data":"5f105c70d549eaaff2e641e557dbade9c0b83960770896c4b9588999ad1426ec"} Jul 10 19:00:36.977948 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.977923 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bxw74" event={"ID":"f9ffcad3-0a20-40c1-9383-454676ea46aa","Type":"ContainerStarted","Data":"66f251c4a4a106c025db684a997500083122c9c91d4a34364685f9c806f8e799"} Jul 10 19:00:36.979561 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.979524 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jb7v6" event={"ID":"ffe36bf2-7d7e-426f-a273-9acdbd6935e4","Type":"ContainerStarted","Data":"f3c901fc5b51a816fe667cbf5eedce92435b16849f97bec2d3f8d7809241ec89"} Jul 10 19:00:36.981006 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.980982 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" event={"ID":"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b","Type":"ContainerStarted","Data":"ed97b67a959db7f27cd6cf247dd0112105b3e4c5babc6466904eb6f726bb1ba5"} Jul 10 19:00:36.982241 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.982212 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gp2jn" event={"ID":"452ded98-132f-4e28-865a-d2f55032625c","Type":"ContainerStarted","Data":"3510e53ada1bf73e15481d4c211e9761f0f9d1148df15a1f75fdeea4ef519aa2"} Jul 10 19:00:36.983428 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.983401 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xczgb" event={"ID":"d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855","Type":"ContainerStarted","Data":"5baf1dac02a3cc348f78d94f0e81ef8383ef4d6e1a48b4609e2eb0e5ab4987c2"} Jul 10 19:00:36.986004 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:36.985961 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d9nf8" event={"ID":"54a84657-5511-4d5f-bc6b-6602b5b2c654","Type":"ContainerStarted","Data":"0ed111494830e4998acc6a0f5ce9ef51fa67e01b89c810c0107382a1f58a0e7f"} Jul 10 19:00:37.461285 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:37.461244 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret\") pod \"global-pull-secret-syncer-j62sj\" (UID: \"2509af5e-c021-4342-b2f3-7448be0ce972\") " pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:37.461493 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:37.461451 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:37.461557 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:37.461519 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret podName:2509af5e-c021-4342-b2f3-7448be0ce972 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:39.461500231 +0000 UTC m=+6.062642367 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret") pod "global-pull-secret-syncer-j62sj" (UID: "2509af5e-c021-4342-b2f3-7448be0ce972") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:37.563442 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:37.562558 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmlp\" (UniqueName: \"kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp\") pod \"network-check-target-gwkcf\" (UID: \"3eb2cd17-af75-4a86-bf49-2ff369e4860d\") " pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:37.563442 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:37.562650 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs\") pod \"network-metrics-daemon-kpm26\" (UID: \"88ca28c8-725b-46f7-becc-ccd04207fba9\") " pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:37.563442 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:37.562805 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:37.563442 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:37.562871 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs podName:88ca28c8-725b-46f7-becc-ccd04207fba9 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:39.562852258 +0000 UTC m=+6.163994377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs") pod "network-metrics-daemon-kpm26" (UID: "88ca28c8-725b-46f7-becc-ccd04207fba9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:37.563442 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:37.563323 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:00:37.563442 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:37.563345 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:00:37.563442 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:37.563358 2561 projected.go:196] Error preparing data for projected volume kube-api-access-qvmlp for pod openshift-network-diagnostics/network-check-target-gwkcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:37.563442 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:37.563404 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp podName:3eb2cd17-af75-4a86-bf49-2ff369e4860d nodeName:}" failed. No retries permitted until 2026-07-10 19:00:39.563389053 +0000 UTC m=+6.164531173 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvmlp" (UniqueName: "kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp") pod "network-check-target-gwkcf" (UID: "3eb2cd17-af75-4a86-bf49-2ff369e4860d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:37.872942 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:37.872890 2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 18:55:36 +0000 UTC" deadline="2028-04-14 16:28:33.911458636 +0000 UTC" Jul 10 19:00:37.873424 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:37.872953 2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15453h27m56.038525596s" Jul 10 19:00:37.958580 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:37.958122 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:37.958580 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:37.958126 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:37.958580 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:37.958248 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwkcf" podUID="3eb2cd17-af75-4a86-bf49-2ff369e4860d" Jul 10 19:00:37.958580 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:37.958288 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:37.958580 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:37.958404 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j62sj" podUID="2509af5e-c021-4342-b2f3-7448be0ce972" Jul 10 19:00:37.958580 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:37.958509 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpm26" podUID="88ca28c8-725b-46f7-becc-ccd04207fba9" Jul 10 19:00:39.017506 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:39.017133 2561 generic.go:356] "Generic (PLEG): container finished" podID="8c924a039fae58f41b47a54ccd1ac6a8" containerID="4bd58ea33a734e378989be0dbc81ef21964bf34875f7243ca62facc28c8ea9a1" exitCode=0 Jul 10 19:00:39.017506 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:39.017331 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal" event={"ID":"8c924a039fae58f41b47a54ccd1ac6a8","Type":"ContainerDied","Data":"4bd58ea33a734e378989be0dbc81ef21964bf34875f7243ca62facc28c8ea9a1"} Jul 10 19:00:39.020499 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:39.020029 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-74.ec2.internal" event={"ID":"7651e5e5e93604167127627171e7b52f","Type":"ContainerStarted","Data":"0d84f4429664b9f11b04318e129258638d855c3501346b145d355828f653ba36"} Jul 10 19:00:39.074275 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:39.073153 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-74.ec2.internal" podStartSLOduration=4.073122474 podStartE2EDuration="4.073122474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:39.072292629 +0000 UTC m=+5.673434765" watchObservedRunningTime="2026-07-10 19:00:39.073122474 +0000 UTC m=+5.674264609" Jul 10 19:00:39.481125 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:39.481041 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret\") pod \"global-pull-secret-syncer-j62sj\" (UID: \"2509af5e-c021-4342-b2f3-7448be0ce972\") " pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:39.481287 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:39.481244 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:39.481362 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:39.481304 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret podName:2509af5e-c021-4342-b2f3-7448be0ce972 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:43.481287033 +0000 UTC m=+10.082429150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret") pod "global-pull-secret-syncer-j62sj" (UID: "2509af5e-c021-4342-b2f3-7448be0ce972") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:39.583244 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:39.582364 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs\") pod \"network-metrics-daemon-kpm26\" (UID: \"88ca28c8-725b-46f7-becc-ccd04207fba9\") " pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:39.583244 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:39.582435 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmlp\" (UniqueName: \"kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp\") pod \"network-check-target-gwkcf\" (UID: \"3eb2cd17-af75-4a86-bf49-2ff369e4860d\") " pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:39.583244 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:39.582588 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:00:39.583244 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:39.582609 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:00:39.583244 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:39.582625 2561 projected.go:196] Error preparing data for projected volume kube-api-access-qvmlp for pod openshift-network-diagnostics/network-check-target-gwkcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:39.583244 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:39.582693 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp podName:3eb2cd17-af75-4a86-bf49-2ff369e4860d nodeName:}" failed. No retries permitted until 2026-07-10 19:00:43.58266441 +0000 UTC m=+10.183806539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvmlp" (UniqueName: "kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp") pod "network-check-target-gwkcf" (UID: "3eb2cd17-af75-4a86-bf49-2ff369e4860d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:39.583244 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:39.583129 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:39.583244 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:39.583208 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs podName:88ca28c8-725b-46f7-becc-ccd04207fba9 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:43.583191456 +0000 UTC m=+10.184333569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs") pod "network-metrics-daemon-kpm26" (UID: "88ca28c8-725b-46f7-becc-ccd04207fba9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:39.959671 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:39.959641 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:39.959848 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:39.959738 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwkcf" podUID="3eb2cd17-af75-4a86-bf49-2ff369e4860d" Jul 10 19:00:39.960384 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:39.960057 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:39.960384 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:39.960143 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j62sj" podUID="2509af5e-c021-4342-b2f3-7448be0ce972" Jul 10 19:00:39.960384 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:39.960152 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:39.960384 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:39.960300 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpm26" podUID="88ca28c8-725b-46f7-becc-ccd04207fba9" Jul 10 19:00:41.957629 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:41.957111 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:41.957629 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:41.957155 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:41.957629 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:41.957123 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:41.957629 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:41.957259 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpm26" podUID="88ca28c8-725b-46f7-becc-ccd04207fba9" Jul 10 19:00:41.957629 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:41.957360 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j62sj" podUID="2509af5e-c021-4342-b2f3-7448be0ce972" Jul 10 19:00:41.957629 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:41.957438 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwkcf" podUID="3eb2cd17-af75-4a86-bf49-2ff369e4860d" Jul 10 19:00:43.518347 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:43.517895 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret\") pod \"global-pull-secret-syncer-j62sj\" (UID: \"2509af5e-c021-4342-b2f3-7448be0ce972\") " pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:43.518347 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:43.518035 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:43.518347 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:43.518082 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret podName:2509af5e-c021-4342-b2f3-7448be0ce972 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:51.518069415 +0000 UTC m=+18.119211528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret") pod "global-pull-secret-syncer-j62sj" (UID: "2509af5e-c021-4342-b2f3-7448be0ce972") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:43.618658 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:43.618611 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmlp\" (UniqueName: \"kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp\") pod \"network-check-target-gwkcf\" (UID: \"3eb2cd17-af75-4a86-bf49-2ff369e4860d\") " pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:43.618853 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:43.618704 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs\") pod \"network-metrics-daemon-kpm26\" (UID: \"88ca28c8-725b-46f7-becc-ccd04207fba9\") " pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:43.618853 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:43.618832 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:43.618853 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:43.618845 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:00:43.619033 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:43.618863 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:00:43.619033 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:43.618876 2561 projected.go:196] Error preparing data for projected volume kube-api-access-qvmlp for pod openshift-network-diagnostics/network-check-target-gwkcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:43.619033 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:43.618902 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs podName:88ca28c8-725b-46f7-becc-ccd04207fba9 nodeName:}" failed. No retries permitted until 2026-07-10 19:00:51.618882918 +0000 UTC m=+18.220025051 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs") pod "network-metrics-daemon-kpm26" (UID: "88ca28c8-725b-46f7-becc-ccd04207fba9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:43.619033 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:43.618918 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp podName:3eb2cd17-af75-4a86-bf49-2ff369e4860d nodeName:}" failed. No retries permitted until 2026-07-10 19:00:51.618909703 +0000 UTC m=+18.220051816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvmlp" (UniqueName: "kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp") pod "network-check-target-gwkcf" (UID: "3eb2cd17-af75-4a86-bf49-2ff369e4860d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:43.959045 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:43.958223 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:43.959045 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:43.958360 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpm26" podUID="88ca28c8-725b-46f7-becc-ccd04207fba9" Jul 10 19:00:43.959045 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:43.958729 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:43.959045 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:43.958800 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j62sj" podUID="2509af5e-c021-4342-b2f3-7448be0ce972" Jul 10 19:00:43.959045 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:43.958874 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:43.959045 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:43.958969 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwkcf" podUID="3eb2cd17-af75-4a86-bf49-2ff369e4860d" Jul 10 19:00:45.958078 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:45.958044 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:45.958564 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:45.958199 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpm26" podUID="88ca28c8-725b-46f7-becc-ccd04207fba9" Jul 10 19:00:45.958647 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:45.958568 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:45.958699 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:45.958650 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j62sj" podUID="2509af5e-c021-4342-b2f3-7448be0ce972" Jul 10 19:00:45.960064 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:45.960044 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:45.960173 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:45.960126 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwkcf" podUID="3eb2cd17-af75-4a86-bf49-2ff369e4860d" Jul 10 19:00:47.957739 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:47.957702 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:47.958217 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:47.957708 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:47.958217 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:47.957828 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpm26" podUID="88ca28c8-725b-46f7-becc-ccd04207fba9" Jul 10 19:00:47.958217 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:47.957842 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:47.958217 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:47.957960 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j62sj" podUID="2509af5e-c021-4342-b2f3-7448be0ce972" Jul 10 19:00:47.958217 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:47.958047 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwkcf" podUID="3eb2cd17-af75-4a86-bf49-2ff369e4860d" Jul 10 19:00:49.957906 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:49.957873 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:49.957906 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:49.957891 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:49.958435 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:49.958009 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwkcf" podUID="3eb2cd17-af75-4a86-bf49-2ff369e4860d" Jul 10 19:00:49.958435 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:49.958108 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpm26" podUID="88ca28c8-725b-46f7-becc-ccd04207fba9" Jul 10 19:00:49.958435 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:49.958162 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:49.958435 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:49.958247 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j62sj" podUID="2509af5e-c021-4342-b2f3-7448be0ce972" Jul 10 19:00:51.576485 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:51.576441 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret\") pod \"global-pull-secret-syncer-j62sj\" (UID: \"2509af5e-c021-4342-b2f3-7448be0ce972\") " pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:51.576928 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:51.576602 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:51.576928 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:51.576659 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret podName:2509af5e-c021-4342-b2f3-7448be0ce972 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:07.576645994 +0000 UTC m=+34.177788108 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret") pod "global-pull-secret-syncer-j62sj" (UID: "2509af5e-c021-4342-b2f3-7448be0ce972") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:00:51.677350 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:51.677297 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs\") pod \"network-metrics-daemon-kpm26\" (UID: \"88ca28c8-725b-46f7-becc-ccd04207fba9\") " pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:51.677535 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:51.677387 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmlp\" (UniqueName: \"kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp\") pod \"network-check-target-gwkcf\" (UID: \"3eb2cd17-af75-4a86-bf49-2ff369e4860d\") " pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:51.677535 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:51.677447 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:51.677672 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:51.677540 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs podName:88ca28c8-725b-46f7-becc-ccd04207fba9 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:07.677519259 +0000 UTC m=+34.278661375 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs") pod "network-metrics-daemon-kpm26" (UID: "88ca28c8-725b-46f7-becc-ccd04207fba9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:00:51.677672 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:51.677543 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:00:51.677672 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:51.677560 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:00:51.677672 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:51.677575 2561 projected.go:196] Error preparing data for projected volume kube-api-access-qvmlp for pod openshift-network-diagnostics/network-check-target-gwkcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:51.677672 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:51.677655 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp podName:3eb2cd17-af75-4a86-bf49-2ff369e4860d nodeName:}" failed. No retries permitted until 2026-07-10 19:01:07.677617759 +0000 UTC m=+34.278759892 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvmlp" (UniqueName: "kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp") pod "network-check-target-gwkcf" (UID: "3eb2cd17-af75-4a86-bf49-2ff369e4860d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:00:51.957236 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:51.957145 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:51.957383 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:51.957145 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:51.957383 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:51.957291 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpm26" podUID="88ca28c8-725b-46f7-becc-ccd04207fba9" Jul 10 19:00:51.957383 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:51.957311 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:51.957536 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:51.957420 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j62sj" podUID="2509af5e-c021-4342-b2f3-7448be0ce972" Jul 10 19:00:51.957536 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:51.957493 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwkcf" podUID="3eb2cd17-af75-4a86-bf49-2ff369e4860d" Jul 10 19:00:53.958232 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:53.958194 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:53.958706 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:53.958305 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:53.958706 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:53.958313 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpm26" podUID="88ca28c8-725b-46f7-becc-ccd04207fba9" Jul 10 19:00:53.958706 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:53.958427 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwkcf" podUID="3eb2cd17-af75-4a86-bf49-2ff369e4860d" Jul 10 19:00:53.958706 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:53.958489 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:53.958706 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:53.958565 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j62sj" podUID="2509af5e-c021-4342-b2f3-7448be0ce972" Jul 10 19:00:55.960619 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:55.960093 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:55.960619 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:55.960427 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpm26" podUID="88ca28c8-725b-46f7-becc-ccd04207fba9" Jul 10 19:00:55.961078 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:55.960858 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:55.961078 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:55.960988 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j62sj" podUID="2509af5e-c021-4342-b2f3-7448be0ce972" Jul 10 19:00:55.961078 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:55.961057 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:55.961259 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:55.961127 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwkcf" podUID="3eb2cd17-af75-4a86-bf49-2ff369e4860d" Jul 10 19:00:56.059038 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:56.059010 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal" event={"ID":"8c924a039fae58f41b47a54ccd1ac6a8","Type":"ContainerStarted","Data":"061c130c051051bec731bcd852865e3174039b77dde7a9eea590448fd97ae4df"} Jul 10 19:00:56.060549 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:56.060498 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" event={"ID":"cfecda89-a83d-44a7-88a9-e053a3a6e311","Type":"ContainerStarted","Data":"07d3cbd681a7ed602f240f864586e32aa5a66c42a8d5059062f475e9b572f065"} Jul 10 19:00:56.062860 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:56.062305 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5w2gc" event={"ID":"4766e5ae-0ae1-4fac-b87b-0b48178ee932","Type":"ContainerStarted","Data":"078e61a617f51f57415641bbd210bae4c4e15d2483510d0daa7d34fdf7ae1298"} Jul 10 19:00:56.063768 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:56.063744 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" event={"ID":"f1365527-36c3-4655-ac1a-eab0b02f92fd","Type":"ContainerStarted","Data":"d30c2577f6d4dd7ceefcb627cbff0ff12b152a5b85414fa79e653ad38cb09450"} Jul 10 19:00:56.064952 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:56.064913 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bxw74" event={"ID":"f9ffcad3-0a20-40c1-9383-454676ea46aa","Type":"ContainerStarted","Data":"560acaef650cb5955afc4097baf25c2d493c3978145533dc3ea994184632ab15"} Jul 10 19:00:56.068850 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:56.068811 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jb7v6" event={"ID":"ffe36bf2-7d7e-426f-a273-9acdbd6935e4","Type":"ContainerStarted","Data":"76ed809d362bb5a52e542bb60733f542d590ff653150796f5516e52df23a9334"} Jul 10 19:00:56.070718 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:56.070691 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" event={"ID":"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b","Type":"ContainerStarted","Data":"88088bf635ed498e4126d7cf8e71d4f92fe4b6742f496a4a28b2680c9b26bb39"} Jul 10 19:00:56.072115 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:56.072094 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gp2jn" event={"ID":"452ded98-132f-4e28-865a-d2f55032625c","Type":"ContainerStarted","Data":"015dd250936590968ad554078300476bda843de3aa393b35cb9d12a5fcc7568b"} Jul 10 19:00:56.073558 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:56.073539 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-d9nf8" event={"ID":"54a84657-5511-4d5f-bc6b-6602b5b2c654","Type":"ContainerStarted","Data":"ec3e9e8351853565780981035c06c687197f2eb4fc7ef0dde0b743d484bb1f3a"} Jul 10 19:00:56.076322 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:56.076283 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-74.ec2.internal" podStartSLOduration=21.076272969 podStartE2EDuration="21.076272969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:56.07604537 +0000 UTC m=+22.677187517" watchObservedRunningTime="2026-07-10 19:00:56.076272969 +0000 UTC m=+22.677415104" Jul 10 19:00:56.095049 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:56.095009 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-d9nf8" podStartSLOduration=2.437027112 podStartE2EDuration="22.094994866s" totalImagesPullingTime="19.657967754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:56.094628743 +0000 UTC m=+22.695770879" watchObservedRunningTime="2026-07-10 19:00:56.094994866 +0000 UTC m=+22.696137002" Jul 10 19:00:56.138044 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:56.137992 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jb7v6" podStartSLOduration=2.522546537 podStartE2EDuration="22.137975818s" totalImagesPullingTime="19.615429281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:56.137476761 +0000 UTC m=+22.738618892" watchObservedRunningTime="2026-07-10 19:00:56.137975818 +0000 UTC m=+22.739117953" Jul 10 19:00:56.156018 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:56.155972 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bxw74" podStartSLOduration=2.5845361220000003 podStartE2EDuration="22.155954176s" totalImagesPullingTime="19.571418054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:56.155523179 +0000 UTC m=+22.756665315" watchObservedRunningTime="2026-07-10 19:00:56.155954176 +0000 UTC m=+22.757096311" Jul 10 19:00:56.201635 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:56.201584 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4c2qf" podStartSLOduration=2.665057088 podStartE2EDuration="22.201566489s" totalImagesPullingTime="19.536509401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:56.197614766 +0000 UTC m=+22.798756901" watchObservedRunningTime="2026-07-10 19:00:56.201566489 +0000 UTC m=+22.802708624" Jul 10 19:00:56.231621 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:56.231571 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gp2jn" podStartSLOduration=2.713862325 podStartE2EDuration="22.231554046s" totalImagesPullingTime="19.517691721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:56.231363358 +0000 UTC m=+22.832505495" watchObservedRunningTime="2026-07-10 19:00:56.231554046 +0000 UTC m=+22.832696181" Jul 10 19:00:56.266495 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:56.266344 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:56.346831 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:56.346801 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:56.444380 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:56.444356 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:57.077660 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.077472 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xczgb" event={"ID":"d49dce93-b7fa-4c9a-b97f-dc3a9ca6b855","Type":"ContainerStarted","Data":"c8cf9738791faddce439854f7f72a2cc8a14e6900869538861caf067ec4b0825"} Jul 10 19:00:57.079018 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.078994 2561 generic.go:356] "Generic (PLEG): container finished" podID="4766e5ae-0ae1-4fac-b87b-0b48178ee932" containerID="078e61a617f51f57415641bbd210bae4c4e15d2483510d0daa7d34fdf7ae1298" exitCode=0 Jul 10 19:00:57.079136 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.079074 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5w2gc" event={"ID":"4766e5ae-0ae1-4fac-b87b-0b48178ee932","Type":"ContainerDied","Data":"078e61a617f51f57415641bbd210bae4c4e15d2483510d0daa7d34fdf7ae1298"} Jul 10 19:00:57.081903 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.081887 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:00:57.082276 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.082256 2561 generic.go:356] "Generic (PLEG): container finished" podID="7c3b76b0-8cb8-489c-ae2e-a8e78982c24b" containerID="c30713112aed2adbf776072317f3a9c786342e077e7f59d8104b69a8dad62e47" exitCode=1 Jul 10 19:00:57.082360 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.082279 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" event={"ID":"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b","Type":"ContainerStarted","Data":"30aee67036d859a3625bc1d77790c0e81288572583c0e64f7be6c00df5f24d43"} Jul 10 19:00:57.082425 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.082368 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" event={"ID":"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b","Type":"ContainerStarted","Data":"d3b0acca7131093b5ea6cd7d66d04fd973ba0901fbe9c1b46ba394e16962d168"} Jul 10 19:00:57.082425 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.082407 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" event={"ID":"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b","Type":"ContainerStarted","Data":"19355d35644824b85c41ed1b828e02ed7a3bfec2a32c605f81170f96ce9f2419"} Jul 10 19:00:57.082495 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.082436 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" event={"ID":"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b","Type":"ContainerStarted","Data":"fce70ebc84730c2c5c3442221e34ee49651ac257588e0198c4e3fee79ac74b61"} Jul 10 19:00:57.082495 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.082471 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" event={"ID":"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b","Type":"ContainerDied","Data":"c30713112aed2adbf776072317f3a9c786342e077e7f59d8104b69a8dad62e47"} Jul 10 19:00:57.097853 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.097812 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xczgb" podStartSLOduration=3.517313104 podStartE2EDuration="23.097799508s" totalImagesPullingTime="19.580486404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:57.097633569 +0000 UTC m=+23.698775703" watchObservedRunningTime="2026-07-10 19:00:57.097799508 +0000 UTC m=+23.698941642" Jul 10 19:00:57.247897 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.247864 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:57.347911 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.347841 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:57.450781 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.450752 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:57.553242 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.553207 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:57.592882 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.592839 2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 19:00:57.653918 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.653872 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:57.897453 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.897225 2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T19:00:57.592857313Z","UUID":"ee62be88-c8fa-4c6e-b545-2f35660117c9","Handler":null,"Name":"","Endpoint":""} Jul 10 19:00:57.899146 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.899119 2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 19:00:57.899288 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.899162 2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 19:00:57.957381 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.957346 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:57.957545 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.957399 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:57.957545 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:57.957408 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:57.957545 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:57.957498 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpm26" podUID="88ca28c8-725b-46f7-becc-ccd04207fba9" Jul 10 19:00:57.957772 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:57.957593 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwkcf" podUID="3eb2cd17-af75-4a86-bf49-2ff369e4860d" Jul 10 19:00:57.957772 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:57.957659 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j62sj" podUID="2509af5e-c021-4342-b2f3-7448be0ce972" Jul 10 19:00:58.048812 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:58.048779 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bxw74" Jul 10 19:00:58.049579 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:58.049557 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bxw74" Jul 10 19:00:58.087257 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:58.087205 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" event={"ID":"f1365527-36c3-4655-ac1a-eab0b02f92fd","Type":"ContainerStarted","Data":"5d19493d2569af9c38c13546e1ea348dec1e413e6474b2a4c007eac35add4449"} Jul 10 19:00:58.087710 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:58.087541 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bxw74" Jul 10 19:00:58.088169 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:58.088152 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bxw74" Jul 10 19:00:58.441299 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:58.441254 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:58.925915 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:58.925688 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:59.026488 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:59.026453 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:59.092124 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:59.092085 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" event={"ID":"f1365527-36c3-4655-ac1a-eab0b02f92fd","Type":"ContainerStarted","Data":"77609d1e6ec5edbd01f60599551bbf1268937ab8e2add70be163f3514755f660"} Jul 10 19:00:59.095603 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:59.095584 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:00:59.096060 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:59.096021 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" event={"ID":"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b","Type":"ContainerStarted","Data":"1ca15ae4ec3f6296524c5ce9d8c7eb179d83d6b08469cecc1926744a3dcf1937"} Jul 10 19:00:59.114213 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:59.114170 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7fb4c" podStartSLOduration=3.120355394 podStartE2EDuration="25.114149594s" totalImagesPullingTime="21.9937942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:00:59.113779783 +0000 UTC m=+25.714921917" watchObservedRunningTime="2026-07-10 19:00:59.114149594 +0000 UTC m=+25.715291730" Jul 10 19:00:59.378999 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:59.378862 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:59.463190 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:59.463155 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:00:59.957803 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:59.957766 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:00:59.957803 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:59.957795 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:00:59.958075 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:00:59.957885 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:00:59.958075 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:59.957909 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwkcf" podUID="3eb2cd17-af75-4a86-bf49-2ff369e4860d" Jul 10 19:00:59.958075 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:59.957984 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpm26" podUID="88ca28c8-725b-46f7-becc-ccd04207fba9" Jul 10 19:00:59.958075 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:00:59.958031 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j62sj" podUID="2509af5e-c021-4342-b2f3-7448be0ce972" Jul 10 19:01:01.282028 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:01.281989 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:01.366596 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:01.366561 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:01.958205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:01.957961 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:01:01.958205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:01.957966 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:01:01.958394 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:01.958214 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpm26" podUID="88ca28c8-725b-46f7-becc-ccd04207fba9" Jul 10 19:01:01.958394 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:01.957970 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:01:01.958394 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:01.958278 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j62sj" podUID="2509af5e-c021-4342-b2f3-7448be0ce972" Jul 10 19:01:01.958394 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:01.958369 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwkcf" podUID="3eb2cd17-af75-4a86-bf49-2ff369e4860d" Jul 10 19:01:02.105324 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:02.105291 2561 generic.go:356] "Generic (PLEG): container finished" podID="4766e5ae-0ae1-4fac-b87b-0b48178ee932" containerID="0286830cf6092ec06053427cf39bf8b57dbec98ca2fecf0b28f6453b1619a4a8" exitCode=0 Jul 10 19:01:02.105496 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:02.105369 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5w2gc" event={"ID":"4766e5ae-0ae1-4fac-b87b-0b48178ee932","Type":"ContainerDied","Data":"0286830cf6092ec06053427cf39bf8b57dbec98ca2fecf0b28f6453b1619a4a8"} Jul 10 19:01:02.108999 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:02.108980 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:01:02.109313 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:02.109292 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" event={"ID":"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b","Type":"ContainerStarted","Data":"29b78a60140d55d9859516c34f0059d33dd4e58d0ba720df1cc44fd61ef68d2d"} Jul 10 19:01:02.109568 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:02.109553 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:01:02.109636 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:02.109625 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:01:02.109714 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:02.109701 2561 scope.go:117] "RemoveContainer" containerID="c30713112aed2adbf776072317f3a9c786342e077e7f59d8104b69a8dad62e47" Jul 10 19:01:02.125774 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:02.125756 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:01:02.820576 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:02.820548 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gp2jn_452ded98-132f-4e28-865a-d2f55032625c/dns-node-resolver/0.log" Jul 10 19:01:03.119317 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:03.119246 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:01:03.119724 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:03.119693 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" event={"ID":"7c3b76b0-8cb8-489c-ae2e-a8e78982c24b","Type":"ContainerStarted","Data":"8f2ee30edfc15c7aabeedded6e7fbe44d540e156455186916ee7227149d16710"} Jul 10 19:01:03.120691 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:03.120658 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:01:03.138962 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:03.138135 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:01:03.149917 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:03.149862 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" podStartSLOduration=9.542226526 podStartE2EDuration="29.149842892s" totalImagesPullingTime="19.607616366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:03.148344506 +0000 UTC m=+29.749486641" watchObservedRunningTime="2026-07-10 19:01:03.149842892 +0000 UTC m=+29.750985029" Jul 10 19:01:03.265478 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:03.265323 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:03.365483 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:03.365451 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:03.425155 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:03.425072 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kpm26"] Jul 10 19:01:03.425318 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:03.425221 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:01:03.425380 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:03.425341 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpm26" podUID="88ca28c8-725b-46f7-becc-ccd04207fba9" Jul 10 19:01:03.427714 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:03.427688 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gwkcf"] Jul 10 19:01:03.427854 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:03.427792 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:01:03.427918 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:03.427886 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwkcf" podUID="3eb2cd17-af75-4a86-bf49-2ff369e4860d" Jul 10 19:01:03.440758 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:03.440730 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j62sj"] Jul 10 19:01:03.440875 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:03.440862 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:01:03.441011 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:03.440989 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j62sj" podUID="2509af5e-c021-4342-b2f3-7448be0ce972" Jul 10 19:01:03.820743 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:03.820719 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jb7v6_ffe36bf2-7d7e-426f-a273-9acdbd6935e4/node-ca/0.log" Jul 10 19:01:04.124202 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:04.124113 2561 generic.go:356] "Generic (PLEG): container finished" podID="4766e5ae-0ae1-4fac-b87b-0b48178ee932" containerID="815c59c3864a10ee91480cfdb7aeaa645f39afcdb3816eaa84e970d1e4d4fce6" exitCode=0 Jul 10 19:01:04.124360 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:04.124198 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5w2gc" event={"ID":"4766e5ae-0ae1-4fac-b87b-0b48178ee932","Type":"ContainerDied","Data":"815c59c3864a10ee91480cfdb7aeaa645f39afcdb3816eaa84e970d1e4d4fce6"} Jul 10 19:01:04.779517 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:04.779477 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:04.850058 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:04.850023 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:04.957709 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:04.957679 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:01:04.957821 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:04.957679 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:01:04.957821 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:04.957808 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpm26" podUID="88ca28c8-725b-46f7-becc-ccd04207fba9" Jul 10 19:01:04.957925 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:04.957688 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:01:04.957925 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:04.957875 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwkcf" podUID="3eb2cd17-af75-4a86-bf49-2ff369e4860d" Jul 10 19:01:04.958052 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:04.957979 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j62sj" podUID="2509af5e-c021-4342-b2f3-7448be0ce972" Jul 10 19:01:05.129065 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:05.128973 2561 generic.go:356] "Generic (PLEG): container finished" podID="4766e5ae-0ae1-4fac-b87b-0b48178ee932" containerID="b92afe4456994ea31f114ccd355c69350562af893d655e9792e86bce08caec21" exitCode=0 Jul 10 19:01:05.129216 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:05.129056 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5w2gc" event={"ID":"4766e5ae-0ae1-4fac-b87b-0b48178ee932","Type":"ContainerDied","Data":"b92afe4456994ea31f114ccd355c69350562af893d655e9792e86bce08caec21"} Jul 10 19:01:05.203481 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:05.203449 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:06.957034 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:06.957003 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:01:06.957873 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:06.957003 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:01:06.957873 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:06.957134 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:01:06.957873 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:06.957139 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j62sj" podUID="2509af5e-c021-4342-b2f3-7448be0ce972" Jul 10 19:01:06.957873 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:06.957222 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gwkcf" podUID="3eb2cd17-af75-4a86-bf49-2ff369e4860d" Jul 10 19:01:06.957873 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:06.957317 2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kpm26" podUID="88ca28c8-725b-46f7-becc-ccd04207fba9" Jul 10 19:01:07.601216 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:07.601178 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret\") pod \"global-pull-secret-syncer-j62sj\" (UID: \"2509af5e-c021-4342-b2f3-7448be0ce972\") " pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:01:07.601389 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:07.601312 2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 19:01:07.601389 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:07.601374 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret podName:2509af5e-c021-4342-b2f3-7448be0ce972 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:39.60135844 +0000 UTC m=+66.202500575 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret") pod "global-pull-secret-syncer-j62sj" (UID: "2509af5e-c021-4342-b2f3-7448be0ce972") : object "kube-system"/"original-pull-secret" not registered Jul 10 19:01:07.702191 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:07.702161 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmlp\" (UniqueName: \"kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp\") pod \"network-check-target-gwkcf\" (UID: \"3eb2cd17-af75-4a86-bf49-2ff369e4860d\") " pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:01:07.702359 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:07.702231 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs\") pod \"network-metrics-daemon-kpm26\" (UID: \"88ca28c8-725b-46f7-becc-ccd04207fba9\") " pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:01:07.702359 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:07.702333 2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:01:07.702359 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:07.702343 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 19:01:07.702538 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:07.702370 2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 19:01:07.702538 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:07.702385 2561 projected.go:196] Error preparing data for projected volume kube-api-access-qvmlp for pod openshift-network-diagnostics/network-check-target-gwkcf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:01:07.702538 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:07.702401 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs podName:88ca28c8-725b-46f7-becc-ccd04207fba9 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:39.702381037 +0000 UTC m=+66.303523151 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs") pod "network-metrics-daemon-kpm26" (UID: "88ca28c8-725b-46f7-becc-ccd04207fba9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 19:01:07.702538 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:07.702428 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp podName:3eb2cd17-af75-4a86-bf49-2ff369e4860d nodeName:}" failed. No retries permitted until 2026-07-10 19:01:39.702416359 +0000 UTC m=+66.303558471 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qvmlp" (UniqueName: "kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp") pod "network-check-target-gwkcf" (UID: "3eb2cd17-af75-4a86-bf49-2ff369e4860d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 19:01:08.672918 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.672841 2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-74.ec2.internal" event="NodeReady" Jul 10 19:01:08.673388 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.673054 2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 19:01:08.724952 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.724898 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-92mhk"] Jul 10 19:01:08.760063 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.760030 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rlz4g"] Jul 10 19:01:08.761078 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.761054 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-92mhk" Jul 10 19:01:08.763812 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.763787 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 19:01:08.764244 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.764221 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jcvlx\"" Jul 10 19:01:08.764830 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.764488 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 19:01:08.764830 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.764560 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 19:01:08.778223 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.778200 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-92mhk"] Jul 10 19:01:08.778316 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.778238 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rlz4g"] Jul 10 19:01:08.778316 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.778272 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hlcx4"] Jul 10 19:01:08.778387 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.778313 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:08.780927 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.780907 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 19:01:08.781084 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.781065 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 19:01:08.781146 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.781135 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 19:01:08.781267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.781250 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 19:01:08.781392 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.781374 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f67rl\"" Jul 10 19:01:08.800438 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.800412 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hlcx4"] Jul 10 19:01:08.800600 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.800532 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:08.803126 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.803105 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 19:01:08.803126 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.803109 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 19:01:08.803292 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.803109 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gmlfz\"" Jul 10 19:01:08.899723 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.899684 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:08.912054 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.912018 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj2xz\" (UniqueName: \"kubernetes.io/projected/a9626c55-cbe1-411e-9996-31d97196f235-kube-api-access-sj2xz\") pod \"dns-default-hlcx4\" (UID: \"a9626c55-cbe1-411e-9996-31d97196f235\") " pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:08.912205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.912084 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vshc8\" (UniqueName: \"kubernetes.io/projected/0d117dad-109a-41fc-a1f0-0530aff4279e-kube-api-access-vshc8\") pod \"ingress-canary-92mhk\" (UID: \"0d117dad-109a-41fc-a1f0-0530aff4279e\") " pod="openshift-ingress-canary/ingress-canary-92mhk" Jul 10 19:01:08.912205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.912136 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/503b2049-91bb-4f7f-a230-6ec6653c6b62-crio-socket\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:08.912205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.912167 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/503b2049-91bb-4f7f-a230-6ec6653c6b62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:08.912205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.912196 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwz8v\" (UniqueName: \"kubernetes.io/projected/503b2049-91bb-4f7f-a230-6ec6653c6b62-kube-api-access-cwz8v\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:08.912383 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.912222 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/503b2049-91bb-4f7f-a230-6ec6653c6b62-data-volume\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:08.912383 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.912249 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9626c55-cbe1-411e-9996-31d97196f235-tmp-dir\") pod \"dns-default-hlcx4\" (UID: \"a9626c55-cbe1-411e-9996-31d97196f235\") " pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:08.912383 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.912345 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9626c55-cbe1-411e-9996-31d97196f235-metrics-tls\") pod \"dns-default-hlcx4\" (UID: \"a9626c55-cbe1-411e-9996-31d97196f235\") " pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:08.912496 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.912379 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9626c55-cbe1-411e-9996-31d97196f235-config-volume\") pod \"dns-default-hlcx4\" (UID: \"a9626c55-cbe1-411e-9996-31d97196f235\") " pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:08.912496 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.912407 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d117dad-109a-41fc-a1f0-0530aff4279e-cert\") pod \"ingress-canary-92mhk\" (UID: \"0d117dad-109a-41fc-a1f0-0530aff4279e\") " pod="openshift-ingress-canary/ingress-canary-92mhk" Jul 10 19:01:08.912496 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.912449 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/503b2049-91bb-4f7f-a230-6ec6653c6b62-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:08.957315 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.957238 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:01:08.957464 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.957241 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:01:08.957464 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.957239 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:01:08.960862 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.960840 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 19:01:08.961006 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.960871 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zb6nn\"" Jul 10 19:01:08.961006 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.960956 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 19:01:08.961125 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.961046 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 19:01:08.961190 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.961171 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sqhv9\"" Jul 10 19:01:08.961249 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:08.961235 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 19:01:09.003768 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.003730 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:09.013681 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.013655 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9626c55-cbe1-411e-9996-31d97196f235-config-volume\") pod \"dns-default-hlcx4\" (UID: \"a9626c55-cbe1-411e-9996-31d97196f235\") " pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:09.013814 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.013692 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d117dad-109a-41fc-a1f0-0530aff4279e-cert\") pod \"ingress-canary-92mhk\" (UID: \"0d117dad-109a-41fc-a1f0-0530aff4279e\") " pod="openshift-ingress-canary/ingress-canary-92mhk" Jul 10 19:01:09.013814 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.013734 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/503b2049-91bb-4f7f-a230-6ec6653c6b62-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:09.013814 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.013761 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj2xz\" (UniqueName: \"kubernetes.io/projected/a9626c55-cbe1-411e-9996-31d97196f235-kube-api-access-sj2xz\") pod \"dns-default-hlcx4\" (UID: \"a9626c55-cbe1-411e-9996-31d97196f235\") " pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:09.013992 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:09.013889 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 19:01:09.013992 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.013930 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vshc8\" (UniqueName: \"kubernetes.io/projected/0d117dad-109a-41fc-a1f0-0530aff4279e-kube-api-access-vshc8\") pod \"ingress-canary-92mhk\" (UID: \"0d117dad-109a-41fc-a1f0-0530aff4279e\") " pod="openshift-ingress-canary/ingress-canary-92mhk" Jul 10 19:01:09.013992 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:09.013982 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/503b2049-91bb-4f7f-a230-6ec6653c6b62-kube-rbac-proxy-cm podName:503b2049-91bb-4f7f-a230-6ec6653c6b62 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:09.513960919 +0000 UTC m=+36.115103034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/503b2049-91bb-4f7f-a230-6ec6653c6b62-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rlz4g" (UID: "503b2049-91bb-4f7f-a230-6ec6653c6b62") : configmap "kube-rbac-proxy" not found Jul 10 19:01:09.014202 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.014043 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/503b2049-91bb-4f7f-a230-6ec6653c6b62-crio-socket\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:09.014202 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.014077 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/503b2049-91bb-4f7f-a230-6ec6653c6b62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:09.014202 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.014111 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwz8v\" (UniqueName: \"kubernetes.io/projected/503b2049-91bb-4f7f-a230-6ec6653c6b62-kube-api-access-cwz8v\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:09.014202 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.014138 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/503b2049-91bb-4f7f-a230-6ec6653c6b62-data-volume\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:09.014202 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.014163 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9626c55-cbe1-411e-9996-31d97196f235-tmp-dir\") pod \"dns-default-hlcx4\" (UID: \"a9626c55-cbe1-411e-9996-31d97196f235\") " pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:09.014461 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.014209 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9626c55-cbe1-411e-9996-31d97196f235-metrics-tls\") pod \"dns-default-hlcx4\" (UID: \"a9626c55-cbe1-411e-9996-31d97196f235\") " pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:09.014461 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.014243 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/503b2049-91bb-4f7f-a230-6ec6653c6b62-crio-socket\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:09.014461 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:09.014320 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 19:01:09.014461 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:09.014347 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 19:01:09.014461 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:09.014365 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503b2049-91bb-4f7f-a230-6ec6653c6b62-insights-runtime-extractor-tls podName:503b2049-91bb-4f7f-a230-6ec6653c6b62 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:09.514351166 +0000 UTC m=+36.115493281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/503b2049-91bb-4f7f-a230-6ec6653c6b62-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rlz4g" (UID: "503b2049-91bb-4f7f-a230-6ec6653c6b62") : secret "insights-runtime-extractor-tls" not found Jul 10 19:01:09.014461 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:09.014391 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9626c55-cbe1-411e-9996-31d97196f235-metrics-tls podName:a9626c55-cbe1-411e-9996-31d97196f235 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:09.514376666 +0000 UTC m=+36.115518781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9626c55-cbe1-411e-9996-31d97196f235-metrics-tls") pod "dns-default-hlcx4" (UID: "a9626c55-cbe1-411e-9996-31d97196f235") : secret "dns-default-metrics-tls" not found Jul 10 19:01:09.014461 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.014382 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a9626c55-cbe1-411e-9996-31d97196f235-config-volume\") pod \"dns-default-hlcx4\" (UID: \"a9626c55-cbe1-411e-9996-31d97196f235\") " pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:09.014778 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.014671 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/503b2049-91bb-4f7f-a230-6ec6653c6b62-data-volume\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:09.014778 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.014679 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a9626c55-cbe1-411e-9996-31d97196f235-tmp-dir\") pod \"dns-default-hlcx4\" (UID: \"a9626c55-cbe1-411e-9996-31d97196f235\") " pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:09.018482 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.018458 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0d117dad-109a-41fc-a1f0-0530aff4279e-cert\") pod \"ingress-canary-92mhk\" (UID: \"0d117dad-109a-41fc-a1f0-0530aff4279e\") " pod="openshift-ingress-canary/ingress-canary-92mhk" Jul 10 19:01:09.024920 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.024894 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vshc8\" (UniqueName: \"kubernetes.io/projected/0d117dad-109a-41fc-a1f0-0530aff4279e-kube-api-access-vshc8\") pod \"ingress-canary-92mhk\" (UID: \"0d117dad-109a-41fc-a1f0-0530aff4279e\") " pod="openshift-ingress-canary/ingress-canary-92mhk" Jul 10 19:01:09.025725 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.025695 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwz8v\" (UniqueName: \"kubernetes.io/projected/503b2049-91bb-4f7f-a230-6ec6653c6b62-kube-api-access-cwz8v\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:09.025868 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.025853 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj2xz\" (UniqueName: \"kubernetes.io/projected/a9626c55-cbe1-411e-9996-31d97196f235-kube-api-access-sj2xz\") pod \"dns-default-hlcx4\" (UID: \"a9626c55-cbe1-411e-9996-31d97196f235\") " pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:09.071808 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.071769 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-92mhk" Jul 10 19:01:09.394743 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.394715 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-92mhk"] Jul 10 19:01:09.461953 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:01:09.461902 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d117dad_109a_41fc_a1f0_0530aff4279e.slice/crio-f93c950d70443c1c82ff1936f8bebffe5d3006b2181fafab328516eae3ed3c37 WatchSource:0}: Error finding container f93c950d70443c1c82ff1936f8bebffe5d3006b2181fafab328516eae3ed3c37: Status 404 returned error can't find the container with id f93c950d70443c1c82ff1936f8bebffe5d3006b2181fafab328516eae3ed3c37 Jul 10 19:01:09.517704 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.517677 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/503b2049-91bb-4f7f-a230-6ec6653c6b62-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:09.517802 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.517760 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/503b2049-91bb-4f7f-a230-6ec6653c6b62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:09.517864 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.517819 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9626c55-cbe1-411e-9996-31d97196f235-metrics-tls\") pod \"dns-default-hlcx4\" (UID: \"a9626c55-cbe1-411e-9996-31d97196f235\") " pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:09.517864 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:09.517831 2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 19:01:09.517979 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:09.517914 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 19:01:09.517979 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:09.517915 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 19:01:09.518083 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:09.517918 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/503b2049-91bb-4f7f-a230-6ec6653c6b62-kube-rbac-proxy-cm podName:503b2049-91bb-4f7f-a230-6ec6653c6b62 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:10.517895951 +0000 UTC m=+37.119038086 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/503b2049-91bb-4f7f-a230-6ec6653c6b62-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rlz4g" (UID: "503b2049-91bb-4f7f-a230-6ec6653c6b62") : configmap "kube-rbac-proxy" not found Jul 10 19:01:09.518083 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:09.518008 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9626c55-cbe1-411e-9996-31d97196f235-metrics-tls podName:a9626c55-cbe1-411e-9996-31d97196f235 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:10.517990228 +0000 UTC m=+37.119132354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9626c55-cbe1-411e-9996-31d97196f235-metrics-tls") pod "dns-default-hlcx4" (UID: "a9626c55-cbe1-411e-9996-31d97196f235") : secret "dns-default-metrics-tls" not found Jul 10 19:01:09.518083 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:09.518029 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503b2049-91bb-4f7f-a230-6ec6653c6b62-insights-runtime-extractor-tls podName:503b2049-91bb-4f7f-a230-6ec6653c6b62 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:10.518015063 +0000 UTC m=+37.119157179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/503b2049-91bb-4f7f-a230-6ec6653c6b62-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rlz4g" (UID: "503b2049-91bb-4f7f-a230-6ec6653c6b62") : secret "insights-runtime-extractor-tls" not found Jul 10 19:01:09.860994 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:09.860887 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:10.024412 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.024382 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-cv526"] Jul 10 19:01:10.041381 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.041354 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-cv526"] Jul 10 19:01:10.041523 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.041484 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-cv526" Jul 10 19:01:10.046510 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.046490 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-btbb9\"" Jul 10 19:01:10.123505 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.123425 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zzhp\" (UniqueName: \"kubernetes.io/projected/275de220-e7da-4573-bd7d-58d19082fb54-kube-api-access-7zzhp\") pod \"network-check-source-869dcfcdcd-cv526\" (UID: \"275de220-e7da-4573-bd7d-58d19082fb54\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-cv526" Jul 10 19:01:10.141878 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.141846 2561 generic.go:356] "Generic (PLEG): container finished" podID="4766e5ae-0ae1-4fac-b87b-0b48178ee932" containerID="381a5b0e7b2e6eebca6fc8ca55235ea1c89a330c9e9e1e3ad55271a69a2c8bf2" exitCode=0 Jul 10 19:01:10.142057 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.141952 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5w2gc" event={"ID":"4766e5ae-0ae1-4fac-b87b-0b48178ee932","Type":"ContainerDied","Data":"381a5b0e7b2e6eebca6fc8ca55235ea1c89a330c9e9e1e3ad55271a69a2c8bf2"} Jul 10 19:01:10.143079 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.143058 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-92mhk" event={"ID":"0d117dad-109a-41fc-a1f0-0530aff4279e","Type":"ContainerStarted","Data":"f93c950d70443c1c82ff1936f8bebffe5d3006b2181fafab328516eae3ed3c37"} Jul 10 19:01:10.224954 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.224905 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zzhp\" (UniqueName: \"kubernetes.io/projected/275de220-e7da-4573-bd7d-58d19082fb54-kube-api-access-7zzhp\") pod \"network-check-source-869dcfcdcd-cv526\" (UID: \"275de220-e7da-4573-bd7d-58d19082fb54\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-cv526" Jul 10 19:01:10.236546 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.236427 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zzhp\" (UniqueName: \"kubernetes.io/projected/275de220-e7da-4573-bd7d-58d19082fb54-kube-api-access-7zzhp\") pod \"network-check-source-869dcfcdcd-cv526\" (UID: \"275de220-e7da-4573-bd7d-58d19082fb54\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-cv526" Jul 10 19:01:10.263483 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.263452 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:10.350436 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.350410 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-cv526" Jul 10 19:01:10.366475 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.366447 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:10.395902 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.395824 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c68b6cf48-lxlf2"] Jul 10 19:01:10.409705 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.409122 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.414756 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.413463 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 19:01:10.414756 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.413714 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 19:01:10.414756 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.414082 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r62lv\"" Jul 10 19:01:10.414756 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.414308 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 19:01:10.425925 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.425857 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c68b6cf48-lxlf2"] Jul 10 19:01:10.425925 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.425890 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 19:01:10.508661 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.508626 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-cv526"] Jul 10 19:01:10.512743 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:01:10.512717 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275de220_e7da_4573_bd7d_58d19082fb54.slice/crio-73b25bfd9d51db3dabecc4c2e7305a400742af8c03d6b1d2ce7a35ff6bb3daa7 WatchSource:0}: Error finding container 73b25bfd9d51db3dabecc4c2e7305a400742af8c03d6b1d2ce7a35ff6bb3daa7: Status 404 returned error can't find the container with id 73b25bfd9d51db3dabecc4c2e7305a400742af8c03d6b1d2ce7a35ff6bb3daa7 Jul 10 19:01:10.526951 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.526906 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-bound-sa-token\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.527077 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.526996 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/503b2049-91bb-4f7f-a230-6ec6653c6b62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:10.527077 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.527027 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtsjk\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-kube-api-access-vtsjk\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.527077 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.527053 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d95a6d04-03d4-4e92-b324-bca98f7ce082-ca-trust-extracted\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.527201 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.527128 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-certificates\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.527201 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:10.527137 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 19:01:10.527201 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.527187 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d95a6d04-03d4-4e92-b324-bca98f7ce082-image-registry-private-configuration\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.527317 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:10.527221 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503b2049-91bb-4f7f-a230-6ec6653c6b62-insights-runtime-extractor-tls podName:503b2049-91bb-4f7f-a230-6ec6653c6b62 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:12.527197256 +0000 UTC m=+39.128339392 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/503b2049-91bb-4f7f-a230-6ec6653c6b62-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rlz4g" (UID: "503b2049-91bb-4f7f-a230-6ec6653c6b62") : secret "insights-runtime-extractor-tls" not found Jul 10 19:01:10.527374 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.527314 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9626c55-cbe1-411e-9996-31d97196f235-metrics-tls\") pod \"dns-default-hlcx4\" (UID: \"a9626c55-cbe1-411e-9996-31d97196f235\") " pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:10.527374 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.527339 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.527374 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.527360 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d95a6d04-03d4-4e92-b324-bca98f7ce082-installation-pull-secrets\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.527556 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.527397 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/503b2049-91bb-4f7f-a230-6ec6653c6b62-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:10.527556 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.527434 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d95a6d04-03d4-4e92-b324-bca98f7ce082-trusted-ca\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.527556 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:10.527450 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 19:01:10.527556 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:10.527504 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9626c55-cbe1-411e-9996-31d97196f235-metrics-tls podName:a9626c55-cbe1-411e-9996-31d97196f235 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:12.527487663 +0000 UTC m=+39.128629793 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9626c55-cbe1-411e-9996-31d97196f235-metrics-tls") pod "dns-default-hlcx4" (UID: "a9626c55-cbe1-411e-9996-31d97196f235") : secret "dns-default-metrics-tls" not found Jul 10 19:01:10.535851 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.535830 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/503b2049-91bb-4f7f-a230-6ec6653c6b62-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:10.628187 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.628151 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d95a6d04-03d4-4e92-b324-bca98f7ce082-trusted-ca\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.628375 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.628212 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-bound-sa-token\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.628375 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.628264 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtsjk\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-kube-api-access-vtsjk\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.628375 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.628291 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d95a6d04-03d4-4e92-b324-bca98f7ce082-ca-trust-extracted\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.628375 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.628308 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-certificates\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.628375 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.628328 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d95a6d04-03d4-4e92-b324-bca98f7ce082-image-registry-private-configuration\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.628632 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.628570 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.628632 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.628623 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d95a6d04-03d4-4e92-b324-bca98f7ce082-installation-pull-secrets\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.628736 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:10.628721 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 19:01:10.628793 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:10.628738 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c68b6cf48-lxlf2: secret "image-registry-tls" not found Jul 10 19:01:10.628793 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.628764 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d95a6d04-03d4-4e92-b324-bca98f7ce082-ca-trust-extracted\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.628793 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:10.628792 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls podName:d95a6d04-03d4-4e92-b324-bca98f7ce082 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:11.12877677 +0000 UTC m=+37.729918885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls") pod "image-registry-c68b6cf48-lxlf2" (UID: "d95a6d04-03d4-4e92-b324-bca98f7ce082") : secret "image-registry-tls" not found Jul 10 19:01:10.629026 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.628998 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-certificates\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.629242 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.629218 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d95a6d04-03d4-4e92-b324-bca98f7ce082-trusted-ca\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.631480 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.631459 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d95a6d04-03d4-4e92-b324-bca98f7ce082-installation-pull-secrets\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.631607 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.631586 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d95a6d04-03d4-4e92-b324-bca98f7ce082-image-registry-private-configuration\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.640065 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.640044 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-bound-sa-token\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:10.640152 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:10.640089 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtsjk\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-kube-api-access-vtsjk\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:11.133338 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:11.133294 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:11.134040 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:11.133469 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 19:01:11.134040 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:11.133490 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c68b6cf48-lxlf2: secret "image-registry-tls" not found Jul 10 19:01:11.134040 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:11.133559 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls podName:d95a6d04-03d4-4e92-b324-bca98f7ce082 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:12.133537044 +0000 UTC m=+38.734679158 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls") pod "image-registry-c68b6cf48-lxlf2" (UID: "d95a6d04-03d4-4e92-b324-bca98f7ce082") : secret "image-registry-tls" not found Jul 10 19:01:11.148900 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:11.148816 2561 generic.go:356] "Generic (PLEG): container finished" podID="4766e5ae-0ae1-4fac-b87b-0b48178ee932" containerID="3f51163ce5c1106b7b68487a551d579ef1080cd64d4dbf4ee5477efbaf89e970" exitCode=0 Jul 10 19:01:11.148900 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:11.148880 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5w2gc" event={"ID":"4766e5ae-0ae1-4fac-b87b-0b48178ee932","Type":"ContainerDied","Data":"3f51163ce5c1106b7b68487a551d579ef1080cd64d4dbf4ee5477efbaf89e970"} Jul 10 19:01:11.150188 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:11.150165 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-cv526" event={"ID":"275de220-e7da-4573-bd7d-58d19082fb54","Type":"ContainerStarted","Data":"73b25bfd9d51db3dabecc4c2e7305a400742af8c03d6b1d2ce7a35ff6bb3daa7"} Jul 10 19:01:11.967149 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:11.967114 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:12.072695 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:12.072663 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:12.144339 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:12.144303 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:12.145039 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:12.144452 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 19:01:12.145039 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:12.144469 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c68b6cf48-lxlf2: secret "image-registry-tls" not found Jul 10 19:01:12.145039 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:12.144524 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls podName:d95a6d04-03d4-4e92-b324-bca98f7ce082 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:14.144507753 +0000 UTC m=+40.745649880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls") pod "image-registry-c68b6cf48-lxlf2" (UID: "d95a6d04-03d4-4e92-b324-bca98f7ce082") : secret "image-registry-tls" not found Jul 10 19:01:12.157678 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:12.157642 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5w2gc" event={"ID":"4766e5ae-0ae1-4fac-b87b-0b48178ee932","Type":"ContainerStarted","Data":"4c9b9537f4f5eebfac722e976b6b21c4ef067d40168c6bf4da8a2dfaa5ab29af"} Jul 10 19:01:12.174541 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:12.174468 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:12.192844 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:12.192789 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5w2gc" podStartSLOduration=7.975167971 podStartE2EDuration="38.192767505s" totalImagesPullingTime="30.217599534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:12.192599395 +0000 UTC m=+38.793741530" watchObservedRunningTime="2026-07-10 19:01:12.192767505 +0000 UTC m=+38.793909643" Jul 10 19:01:12.371504 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:12.371466 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:12.452610 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:12.452557 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:12.547958 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:12.547907 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9626c55-cbe1-411e-9996-31d97196f235-metrics-tls\") pod \"dns-default-hlcx4\" (UID: \"a9626c55-cbe1-411e-9996-31d97196f235\") " pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:12.548120 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:12.548031 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/503b2049-91bb-4f7f-a230-6ec6653c6b62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:12.548120 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:12.548090 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 19:01:12.548229 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:12.548158 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 19:01:12.548229 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:12.548167 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9626c55-cbe1-411e-9996-31d97196f235-metrics-tls podName:a9626c55-cbe1-411e-9996-31d97196f235 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:16.548145748 +0000 UTC m=+43.149287874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9626c55-cbe1-411e-9996-31d97196f235-metrics-tls") pod "dns-default-hlcx4" (UID: "a9626c55-cbe1-411e-9996-31d97196f235") : secret "dns-default-metrics-tls" not found Jul 10 19:01:12.548229 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:12.548204 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503b2049-91bb-4f7f-a230-6ec6653c6b62-insights-runtime-extractor-tls podName:503b2049-91bb-4f7f-a230-6ec6653c6b62 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:16.548188863 +0000 UTC m=+43.149330977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/503b2049-91bb-4f7f-a230-6ec6653c6b62-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rlz4g" (UID: "503b2049-91bb-4f7f-a230-6ec6653c6b62") : secret "insights-runtime-extractor-tls" not found Jul 10 19:01:12.703280 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:12.703110 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:12.800902 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:12.800864 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:13.067420 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:13.067378 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nfcpl"] Jul 10 19:01:13.081963 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:13.081918 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nfcpl" Jul 10 19:01:13.083300 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:13.083255 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nfcpl"] Jul 10 19:01:13.084777 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:13.084712 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 10 19:01:13.085510 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:13.085492 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 10 19:01:13.085510 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:13.085502 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8njds\"" Jul 10 19:01:13.130032 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:13.129794 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:13.161998 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:13.161928 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-92mhk" event={"ID":"0d117dad-109a-41fc-a1f0-0530aff4279e","Type":"ContainerStarted","Data":"5788e6995cebc60e797518923858e76fcea858ac8daef09e6580595c646c36b5"} Jul 10 19:01:13.229083 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:13.228995 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:13.254421 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:13.254375 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h26lh\" (UniqueName: \"kubernetes.io/projected/06cdb497-0536-4b4e-99b1-b950067a5fa8-kube-api-access-h26lh\") pod \"migrator-7f59b54f6c-nfcpl\" (UID: \"06cdb497-0536-4b4e-99b1-b950067a5fa8\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nfcpl" Jul 10 19:01:13.355400 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:13.355266 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h26lh\" (UniqueName: \"kubernetes.io/projected/06cdb497-0536-4b4e-99b1-b950067a5fa8-kube-api-access-h26lh\") pod \"migrator-7f59b54f6c-nfcpl\" (UID: \"06cdb497-0536-4b4e-99b1-b950067a5fa8\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nfcpl" Jul 10 19:01:13.365955 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:13.365903 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h26lh\" (UniqueName: \"kubernetes.io/projected/06cdb497-0536-4b4e-99b1-b950067a5fa8-kube-api-access-h26lh\") pod \"migrator-7f59b54f6c-nfcpl\" (UID: \"06cdb497-0536-4b4e-99b1-b950067a5fa8\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nfcpl" Jul 10 19:01:13.393902 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:13.393870 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nfcpl" Jul 10 19:01:13.788627 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:13.788574 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-92mhk" podStartSLOduration=2.785100589 podStartE2EDuration="5.788553281s" totalImagesPullingTime="3.003452692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:13.182966432 +0000 UTC m=+39.784108568" watchObservedRunningTime="2026-07-10 19:01:13.788553281 +0000 UTC m=+40.389695417" Jul 10 19:01:13.788987 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:13.788968 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nfcpl"] Jul 10 19:01:13.791532 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:01:13.791505 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06cdb497_0536_4b4e_99b1_b950067a5fa8.slice/crio-a75dd00ed67ea7408cc44cd80731e6cc2a44af07b4963e14b4af4b5cadf9496d WatchSource:0}: Error finding container a75dd00ed67ea7408cc44cd80731e6cc2a44af07b4963e14b4af4b5cadf9496d: Status 404 returned error can't find the container with id a75dd00ed67ea7408cc44cd80731e6cc2a44af07b4963e14b4af4b5cadf9496d Jul 10 19:01:14.161476 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.161451 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:14.161652 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:14.161633 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 19:01:14.161695 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:14.161657 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c68b6cf48-lxlf2: secret "image-registry-tls" not found Jul 10 19:01:14.161728 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:14.161721 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls podName:d95a6d04-03d4-4e92-b324-bca98f7ce082 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:18.161700765 +0000 UTC m=+44.762842893 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls") pod "image-registry-c68b6cf48-lxlf2" (UID: "d95a6d04-03d4-4e92-b324-bca98f7ce082") : secret "image-registry-tls" not found Jul 10 19:01:14.165759 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.165731 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-cv526" event={"ID":"275de220-e7da-4573-bd7d-58d19082fb54","Type":"ContainerStarted","Data":"328e236a89d6a1b9692f73ed21df259aa34ee307c1f069386f8f0951600eed97"} Jul 10 19:01:14.166825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.166793 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nfcpl" event={"ID":"06cdb497-0536-4b4e-99b1-b950067a5fa8","Type":"ContainerStarted","Data":"a75dd00ed67ea7408cc44cd80731e6cc2a44af07b4963e14b4af4b5cadf9496d"} Jul 10 19:01:14.269272 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.269235 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-2b65l"] Jul 10 19:01:14.293396 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.293370 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-2b65l"] Jul 10 19:01:14.293548 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.293465 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-2b65l" Jul 10 19:01:14.296297 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.296257 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rbrpl\"" Jul 10 19:01:14.296297 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.296270 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 10 19:01:14.296476 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.296360 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 10 19:01:14.297158 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.297128 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 10 19:01:14.297158 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.297143 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 10 19:01:14.339412 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.339364 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:14.414417 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.414388 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:14.464592 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.464551 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e52f2cb0-823a-42b2-ab53-d3e7249ea04c-signing-key\") pod \"service-ca-79f68b9f4f-2b65l\" (UID: \"e52f2cb0-823a-42b2-ab53-d3e7249ea04c\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2b65l" Jul 10 19:01:14.464786 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.464602 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgphc\" (UniqueName: \"kubernetes.io/projected/e52f2cb0-823a-42b2-ab53-d3e7249ea04c-kube-api-access-jgphc\") pod \"service-ca-79f68b9f4f-2b65l\" (UID: \"e52f2cb0-823a-42b2-ab53-d3e7249ea04c\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2b65l" Jul 10 19:01:14.464786 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.464769 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e52f2cb0-823a-42b2-ab53-d3e7249ea04c-signing-cabundle\") pod \"service-ca-79f68b9f4f-2b65l\" (UID: \"e52f2cb0-823a-42b2-ab53-d3e7249ea04c\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2b65l" Jul 10 19:01:14.565889 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.565809 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e52f2cb0-823a-42b2-ab53-d3e7249ea04c-signing-cabundle\") pod \"service-ca-79f68b9f4f-2b65l\" (UID: \"e52f2cb0-823a-42b2-ab53-d3e7249ea04c\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2b65l" Jul 10 19:01:14.565889 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.565846 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e52f2cb0-823a-42b2-ab53-d3e7249ea04c-signing-key\") pod \"service-ca-79f68b9f4f-2b65l\" (UID: \"e52f2cb0-823a-42b2-ab53-d3e7249ea04c\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2b65l" Jul 10 19:01:14.565889 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.565866 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgphc\" (UniqueName: \"kubernetes.io/projected/e52f2cb0-823a-42b2-ab53-d3e7249ea04c-kube-api-access-jgphc\") pod \"service-ca-79f68b9f4f-2b65l\" (UID: \"e52f2cb0-823a-42b2-ab53-d3e7249ea04c\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2b65l" Jul 10 19:01:14.566553 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.566524 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e52f2cb0-823a-42b2-ab53-d3e7249ea04c-signing-cabundle\") pod \"service-ca-79f68b9f4f-2b65l\" (UID: \"e52f2cb0-823a-42b2-ab53-d3e7249ea04c\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2b65l" Jul 10 19:01:14.569486 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.569463 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e52f2cb0-823a-42b2-ab53-d3e7249ea04c-signing-key\") pod \"service-ca-79f68b9f4f-2b65l\" (UID: \"e52f2cb0-823a-42b2-ab53-d3e7249ea04c\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2b65l" Jul 10 19:01:14.575288 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.575265 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgphc\" (UniqueName: \"kubernetes.io/projected/e52f2cb0-823a-42b2-ab53-d3e7249ea04c-kube-api-access-jgphc\") pod \"service-ca-79f68b9f4f-2b65l\" (UID: \"e52f2cb0-823a-42b2-ab53-d3e7249ea04c\") " pod="openshift-service-ca/service-ca-79f68b9f4f-2b65l" Jul 10 19:01:14.603251 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.603227 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-2b65l" Jul 10 19:01:14.760127 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:14.759960 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-2b65l"] Jul 10 19:01:14.763669 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:01:14.763624 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode52f2cb0_823a_42b2_ab53_d3e7249ea04c.slice/crio-ea6f4c43d6dfd420cd1987934f92ee516858f23c2ecd7e3a9f91dc37475aa4e7 WatchSource:0}: Error finding container ea6f4c43d6dfd420cd1987934f92ee516858f23c2ecd7e3a9f91dc37475aa4e7: Status 404 returned error can't find the container with id ea6f4c43d6dfd420cd1987934f92ee516858f23c2ecd7e3a9f91dc37475aa4e7 Jul 10 19:01:15.170523 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:15.170483 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-2b65l" event={"ID":"e52f2cb0-823a-42b2-ab53-d3e7249ea04c","Type":"ContainerStarted","Data":"ea6f4c43d6dfd420cd1987934f92ee516858f23c2ecd7e3a9f91dc37475aa4e7"} Jul 10 19:01:15.189107 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:15.189053 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-cv526" podStartSLOduration=1.654372003 podStartE2EDuration="5.18903653s" totalImagesPullingTime="3.534664527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:15.188198106 +0000 UTC m=+41.789340251" watchObservedRunningTime="2026-07-10 19:01:15.18903653 +0000 UTC m=+41.790178666" Jul 10 19:01:15.272863 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:15.272827 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:15.976003 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:15.975903 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:16.077440 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:16.077406 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:16.174237 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:16.174204 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nfcpl" event={"ID":"06cdb497-0536-4b4e-99b1-b950067a5fa8","Type":"ContainerStarted","Data":"9058a4f8e117a01acfe0908372036413b029d3d5337961300a1da3e1aa96f955"} Jul 10 19:01:16.174732 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:16.174260 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nfcpl" event={"ID":"06cdb497-0536-4b4e-99b1-b950067a5fa8","Type":"ContainerStarted","Data":"e733886ab48c1a85e466220ed9a938ff4b8b22fd0025daddf4f164c7e39128b7"} Jul 10 19:01:16.194602 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:16.194552 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-nfcpl" podStartSLOduration=1.391343332 podStartE2EDuration="3.194537575s" totalImagesPullingTime="1.803194243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:16.193370702 +0000 UTC m=+42.794512837" watchObservedRunningTime="2026-07-10 19:01:16.194537575 +0000 UTC m=+42.795679709" Jul 10 19:01:16.580285 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:16.580245 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/503b2049-91bb-4f7f-a230-6ec6653c6b62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:16.580475 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:16.580307 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9626c55-cbe1-411e-9996-31d97196f235-metrics-tls\") pod \"dns-default-hlcx4\" (UID: \"a9626c55-cbe1-411e-9996-31d97196f235\") " pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:16.580475 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:16.580402 2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 19:01:16.580475 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:16.580404 2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 19:01:16.580475 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:16.580454 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a9626c55-cbe1-411e-9996-31d97196f235-metrics-tls podName:a9626c55-cbe1-411e-9996-31d97196f235 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:24.580442446 +0000 UTC m=+51.181584559 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a9626c55-cbe1-411e-9996-31d97196f235-metrics-tls") pod "dns-default-hlcx4" (UID: "a9626c55-cbe1-411e-9996-31d97196f235") : secret "dns-default-metrics-tls" not found Jul 10 19:01:16.580475 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:16.580468 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/503b2049-91bb-4f7f-a230-6ec6653c6b62-insights-runtime-extractor-tls podName:503b2049-91bb-4f7f-a230-6ec6653c6b62 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:24.580462511 +0000 UTC m=+51.181604624 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/503b2049-91bb-4f7f-a230-6ec6653c6b62-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rlz4g" (UID: "503b2049-91bb-4f7f-a230-6ec6653c6b62") : secret "insights-runtime-extractor-tls" not found Jul 10 19:01:18.181447 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:18.181355 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-2b65l" event={"ID":"e52f2cb0-823a-42b2-ab53-d3e7249ea04c","Type":"ContainerStarted","Data":"cbbdc67e8dd705c6bef55c818a4d8df0d220a95cefecd892f54ea48d88669a0c"} Jul 10 19:01:18.194852 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:18.194820 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:18.195012 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:18.194945 2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 19:01:18.195012 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:18.194963 2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-c68b6cf48-lxlf2: secret "image-registry-tls" not found Jul 10 19:01:18.195110 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:01:18.195013 2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls podName:d95a6d04-03d4-4e92-b324-bca98f7ce082 nodeName:}" failed. No retries permitted until 2026-07-10 19:01:26.194998614 +0000 UTC m=+52.796140758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls") pod "image-registry-c68b6cf48-lxlf2" (UID: "d95a6d04-03d4-4e92-b324-bca98f7ce082") : secret "image-registry-tls" not found Jul 10 19:01:18.202863 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:18.202824 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-2b65l" podStartSLOduration=1.1301796259999999 podStartE2EDuration="4.202801347s" totalImagesPullingTime="3.072621721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:18.202144651 +0000 UTC m=+44.803286785" watchObservedRunningTime="2026-07-10 19:01:18.202801347 +0000 UTC m=+44.803943516" Jul 10 19:01:18.760176 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:18.760138 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:18.933642 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:18.933611 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:19.034122 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:19.034024 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:19.134110 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:19.134079 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:19.234764 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:19.234730 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:19.453200 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:19.453112 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:20.255656 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:20.255620 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:20.646119 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:20.646042 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:21.057595 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:21.057553 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:21.246525 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:21.246498 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:21.848023 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:21.847990 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:22.042961 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:22.042912 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:22.456036 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:22.456004 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:22.523055 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:22.523011 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:22.940079 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:22.940048 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:23.050242 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:23.050210 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:23.642901 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:23.642865 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:23.711956 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:23.711907 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:23.842431 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:23.842397 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:24.243676 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:24.243640 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:24.452598 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:24.452565 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:24.646793 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:24.646703 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/503b2049-91bb-4f7f-a230-6ec6653c6b62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:24.646793 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:24.646755 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9626c55-cbe1-411e-9996-31d97196f235-metrics-tls\") pod \"dns-default-hlcx4\" (UID: \"a9626c55-cbe1-411e-9996-31d97196f235\") " pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:24.649745 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:24.649718 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a9626c55-cbe1-411e-9996-31d97196f235-metrics-tls\") pod \"dns-default-hlcx4\" (UID: \"a9626c55-cbe1-411e-9996-31d97196f235\") " pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:24.649884 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:24.649864 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/503b2049-91bb-4f7f-a230-6ec6653c6b62-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rlz4g\" (UID: \"503b2049-91bb-4f7f-a230-6ec6653c6b62\") " pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:24.687502 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:24.687467 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rlz4g" Jul 10 19:01:24.710683 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:24.710653 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:24.846358 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:24.846308 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rlz4g"] Jul 10 19:01:24.849245 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:01:24.849216 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod503b2049_91bb_4f7f_a230_6ec6653c6b62.slice/crio-3715ec5fffed217307698ae347bc047ca89c5c072247f6ddb2bfe6534eb2b9ea WatchSource:0}: Error finding container 3715ec5fffed217307698ae347bc047ca89c5c072247f6ddb2bfe6534eb2b9ea: Status 404 returned error can't find the container with id 3715ec5fffed217307698ae347bc047ca89c5c072247f6ddb2bfe6534eb2b9ea Jul 10 19:01:24.850747 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:24.850720 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:24.867591 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:24.867567 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hlcx4"] Jul 10 19:01:24.870292 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:01:24.870261 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9626c55_cbe1_411e_9996_31d97196f235.slice/crio-e781aeced75163a7849fe2217b563e214c2ff5dbb840c25c8d0713e02d65788b WatchSource:0}: Error finding container e781aeced75163a7849fe2217b563e214c2ff5dbb840c25c8d0713e02d65788b: Status 404 returned error can't find the container with id e781aeced75163a7849fe2217b563e214c2ff5dbb840c25c8d0713e02d65788b Jul 10 19:01:25.128190 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:25.128159 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:25.211685 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:25.211553 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hlcx4" event={"ID":"a9626c55-cbe1-411e-9996-31d97196f235","Type":"ContainerStarted","Data":"e781aeced75163a7849fe2217b563e214c2ff5dbb840c25c8d0713e02d65788b"} Jul 10 19:01:25.216442 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:25.216275 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rlz4g" event={"ID":"503b2049-91bb-4f7f-a230-6ec6653c6b62","Type":"ContainerStarted","Data":"3715ec5fffed217307698ae347bc047ca89c5c072247f6ddb2bfe6534eb2b9ea"} Jul 10 19:01:25.452895 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:25.452808 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:25.848340 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:25.848186 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:26.050461 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:26.050396 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:26.261888 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:26.261849 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:26.266901 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:26.266839 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls\") pod \"image-registry-c68b6cf48-lxlf2\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:26.329200 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:26.329165 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:26.459811 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:26.459672 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:26.517367 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:26.517284 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c68b6cf48-lxlf2"] Jul 10 19:01:26.523075 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:01:26.523040 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95a6d04_03d4_4e92_b324_bca98f7ce082.slice/crio-f6a92bf610dee666b352f35066c6ecd56e93395896aae56c57fe9a1c35f15ff1 WatchSource:0}: Error finding container f6a92bf610dee666b352f35066c6ecd56e93395896aae56c57fe9a1c35f15ff1: Status 404 returned error can't find the container with id f6a92bf610dee666b352f35066c6ecd56e93395896aae56c57fe9a1c35f15ff1 Jul 10 19:01:26.968480 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:26.968351 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:27.069086 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:27.069051 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:27.226889 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:27.226798 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" event={"ID":"d95a6d04-03d4-4e92-b324-bca98f7ce082","Type":"ContainerStarted","Data":"880bb2783072bd5b003d28a4b6ba5c1fa0a59dd7e6911783070b5da0b58bf35e"} Jul 10 19:01:27.226889 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:27.226870 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" event={"ID":"d95a6d04-03d4-4e92-b324-bca98f7ce082","Type":"ContainerStarted","Data":"f6a92bf610dee666b352f35066c6ecd56e93395896aae56c57fe9a1c35f15ff1"} Jul 10 19:01:27.227555 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:27.227530 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:27.249878 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:27.249829 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" podStartSLOduration=17.249812317 podStartE2EDuration="17.249812317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:27.248549352 +0000 UTC m=+53.849691488" watchObservedRunningTime="2026-07-10 19:01:27.249812317 +0000 UTC m=+53.850954451" Jul 10 19:01:27.297154 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:27.297116 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:27.426541 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:27.426507 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:27.526959 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:27.526899 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:27.594765 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:27.594731 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:27.731456 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:27.731409 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:27.842485 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:27.842408 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:28.061822 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:28.061789 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:28.156352 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:28.156214 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:28.278443 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:28.278346 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:28.509979 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:28.509927 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:28.609130 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:28.609068 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:28.874594 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:28.874471 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:28.971891 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:28.971854 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:29.072694 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:29.072658 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:29.173126 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:29.173039 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:29.664560 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:29.664440 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:29.847919 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:29.847832 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:30.239246 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:30.239218 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hlcx4" event={"ID":"a9626c55-cbe1-411e-9996-31d97196f235","Type":"ContainerStarted","Data":"fe8ca699e27fb4b6aa90feb2a3774b05bdad06f193ecf40d71d0507bf2ebc722"} Jul 10 19:01:30.239349 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:30.239282 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hlcx4" event={"ID":"a9626c55-cbe1-411e-9996-31d97196f235","Type":"ContainerStarted","Data":"8b5ca0b8e43fa4194e92471af818fe1a43e2bff0eb811221a810313000db1f2e"} Jul 10 19:01:30.239349 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:30.239317 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:30.240699 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:30.240678 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rlz4g" event={"ID":"503b2049-91bb-4f7f-a230-6ec6653c6b62","Type":"ContainerStarted","Data":"137f609c2073d7e943fb27d80f9258723b20f75ed58fd746558fcf18af146820"} Jul 10 19:01:30.262434 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:30.262388 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hlcx4" podStartSLOduration=17.601785015 podStartE2EDuration="22.262372579s" totalImagesPullingTime="4.660587564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:30.261717907 +0000 UTC m=+56.862860043" watchObservedRunningTime="2026-07-10 19:01:30.262372579 +0000 UTC m=+56.863514691" Jul 10 19:01:30.560180 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:30.560134 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:30.682792 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:30.682761 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:31.246022 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:31.245983 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rlz4g" event={"ID":"503b2049-91bb-4f7f-a230-6ec6653c6b62","Type":"ContainerStarted","Data":"b8cf13fe36634dcb51a817ae8f620f9cf270f0efa434660c1267b88b768e5392"} Jul 10 19:01:32.507265 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:32.507225 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:32.612591 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:32.612493 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:32.713407 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:32.713352 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:33.025953 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:33.025909 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:33.255230 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:33.255192 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rlz4g" event={"ID":"503b2049-91bb-4f7f-a230-6ec6653c6b62","Type":"ContainerStarted","Data":"e970531087db1bb422b4e1e2ccd46e235b6c435d2b5d4bf14e4f514e6a9c59f8"} Jul 10 19:01:34.425151 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.425094 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rlz4g" podStartSLOduration=18.342834975 podStartE2EDuration="26.425075976s" totalImagesPullingTime="8.082241001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:01:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:33.27887185 +0000 UTC m=+59.880013986" watchObservedRunningTime="2026-07-10 19:01:34.425075976 +0000 UTC m=+61.026218108" Jul 10 19:01:34.425535 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.425239 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn"] Jul 10 19:01:34.428158 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.428142 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.440314 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.440289 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 10 19:01:34.440515 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.440502 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 19:01:34.440578 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.440526 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 10 19:01:34.440578 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.440553 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 19:01:34.440687 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.440618 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 10 19:01:34.441328 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.441311 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 19:01:34.441574 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.441557 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 10 19:01:34.450440 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.450412 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn"] Jul 10 19:01:34.532790 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.532748 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/144091a1-453a-457b-a7d9-52038e53d0ed-hub\") pod \"cluster-proxy-proxy-agent-5f5cbc9b88-s56sn\" (UID: \"144091a1-453a-457b-a7d9-52038e53d0ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.532983 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.532872 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/144091a1-453a-457b-a7d9-52038e53d0ed-ca\") pod \"cluster-proxy-proxy-agent-5f5cbc9b88-s56sn\" (UID: \"144091a1-453a-457b-a7d9-52038e53d0ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.532983 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.532901 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/144091a1-453a-457b-a7d9-52038e53d0ed-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5f5cbc9b88-s56sn\" (UID: \"144091a1-453a-457b-a7d9-52038e53d0ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.532983 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.532921 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/144091a1-453a-457b-a7d9-52038e53d0ed-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5f5cbc9b88-s56sn\" (UID: \"144091a1-453a-457b-a7d9-52038e53d0ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.532983 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.532964 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/144091a1-453a-457b-a7d9-52038e53d0ed-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5f5cbc9b88-s56sn\" (UID: \"144091a1-453a-457b-a7d9-52038e53d0ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.533186 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.533027 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cgxd\" (UniqueName: \"kubernetes.io/projected/144091a1-453a-457b-a7d9-52038e53d0ed-kube-api-access-6cgxd\") pod \"cluster-proxy-proxy-agent-5f5cbc9b88-s56sn\" (UID: \"144091a1-453a-457b-a7d9-52038e53d0ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.563776 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.563741 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c68b6cf48-lxlf2"] Jul 10 19:01:34.633538 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.633507 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/144091a1-453a-457b-a7d9-52038e53d0ed-hub\") pod \"cluster-proxy-proxy-agent-5f5cbc9b88-s56sn\" (UID: \"144091a1-453a-457b-a7d9-52038e53d0ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.633751 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.633591 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/144091a1-453a-457b-a7d9-52038e53d0ed-ca\") pod \"cluster-proxy-proxy-agent-5f5cbc9b88-s56sn\" (UID: \"144091a1-453a-457b-a7d9-52038e53d0ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.633751 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.633611 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/144091a1-453a-457b-a7d9-52038e53d0ed-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5f5cbc9b88-s56sn\" (UID: \"144091a1-453a-457b-a7d9-52038e53d0ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.633751 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.633626 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/144091a1-453a-457b-a7d9-52038e53d0ed-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5f5cbc9b88-s56sn\" (UID: \"144091a1-453a-457b-a7d9-52038e53d0ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.633751 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.633643 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/144091a1-453a-457b-a7d9-52038e53d0ed-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5f5cbc9b88-s56sn\" (UID: \"144091a1-453a-457b-a7d9-52038e53d0ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.633751 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.633663 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cgxd\" (UniqueName: \"kubernetes.io/projected/144091a1-453a-457b-a7d9-52038e53d0ed-kube-api-access-6cgxd\") pod \"cluster-proxy-proxy-agent-5f5cbc9b88-s56sn\" (UID: \"144091a1-453a-457b-a7d9-52038e53d0ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.634555 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.634457 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/144091a1-453a-457b-a7d9-52038e53d0ed-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5f5cbc9b88-s56sn\" (UID: \"144091a1-453a-457b-a7d9-52038e53d0ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.636206 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.636180 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/144091a1-453a-457b-a7d9-52038e53d0ed-ca\") pod \"cluster-proxy-proxy-agent-5f5cbc9b88-s56sn\" (UID: \"144091a1-453a-457b-a7d9-52038e53d0ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.636350 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.636328 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/144091a1-453a-457b-a7d9-52038e53d0ed-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5f5cbc9b88-s56sn\" (UID: \"144091a1-453a-457b-a7d9-52038e53d0ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.636624 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.636606 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/144091a1-453a-457b-a7d9-52038e53d0ed-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5f5cbc9b88-s56sn\" (UID: \"144091a1-453a-457b-a7d9-52038e53d0ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.636689 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.636607 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/144091a1-453a-457b-a7d9-52038e53d0ed-hub\") pod \"cluster-proxy-proxy-agent-5f5cbc9b88-s56sn\" (UID: \"144091a1-453a-457b-a7d9-52038e53d0ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.647193 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.647170 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cgxd\" (UniqueName: \"kubernetes.io/projected/144091a1-453a-457b-a7d9-52038e53d0ed-kube-api-access-6cgxd\") pod \"cluster-proxy-proxy-agent-5f5cbc9b88-s56sn\" (UID: \"144091a1-453a-457b-a7d9-52038e53d0ed\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.704194 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.704101 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:34.751614 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.751571 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" Jul 10 19:01:34.849705 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.849666 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:34.894585 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.894533 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn"] Jul 10 19:01:34.897838 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:01:34.897814 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod144091a1_453a_457b_a7d9_52038e53d0ed.slice/crio-9906a659d647b0efa808656a1db76b2c4949a3a7b49a8c55ca342c546c24270f WatchSource:0}: Error finding container 9906a659d647b0efa808656a1db76b2c4949a3a7b49a8c55ca342c546c24270f: Status 404 returned error can't find the container with id 9906a659d647b0efa808656a1db76b2c4949a3a7b49a8c55ca342c546c24270f Jul 10 19:01:34.987655 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:34.987565 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:35.127373 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:35.127334 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:35.140772 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:35.140742 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-knrq9" Jul 10 19:01:35.238875 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:35.238787 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:35.262533 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:35.262498 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" event={"ID":"144091a1-453a-457b-a7d9-52038e53d0ed","Type":"ContainerStarted","Data":"9906a659d647b0efa808656a1db76b2c4949a3a7b49a8c55ca342c546c24270f"} Jul 10 19:01:35.517733 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:35.517703 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:35.596072 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:35.596043 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:35.919149 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:35.919061 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:35.997563 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:35.997523 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:38.447103 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:38.447059 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:38.549100 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:38.549063 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:38.780192 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:38.780158 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:38.899912 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:38.899884 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:39.277447 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:39.277410 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" event={"ID":"144091a1-453a-457b-a7d9-52038e53d0ed","Type":"ContainerStarted","Data":"c576c446f0734379008058527250035a3c573ef7fc4cbe04fa62c1cecfc7a1ea"} Jul 10 19:01:39.277447 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:39.277449 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" event={"ID":"144091a1-453a-457b-a7d9-52038e53d0ed","Type":"ContainerStarted","Data":"7d662a4f50c8bcef13f84f8604ee6c021af4aa23822250c201425915fe567ed4"} Jul 10 19:01:39.277693 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:39.277464 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" event={"ID":"144091a1-453a-457b-a7d9-52038e53d0ed","Type":"ContainerStarted","Data":"8d8cd2dd14b9e8bd46311837dc165ffaa17bc5ccbac6a54ce70b5010ce2d5d4b"} Jul 10 19:01:39.302047 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:39.302001 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5f5cbc9b88-s56sn" podStartSLOduration=1.4593959380000001 podStartE2EDuration="5.301987052s" totalImagesPullingTime="3.842591114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:39.301844728 +0000 UTC m=+65.902986860" watchObservedRunningTime="2026-07-10 19:01:39.301987052 +0000 UTC m=+65.903129184" Jul 10 19:01:39.681743 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:39.681648 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret\") pod \"global-pull-secret-syncer-j62sj\" (UID: \"2509af5e-c021-4342-b2f3-7448be0ce972\") " pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:01:39.684506 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:39.684484 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 19:01:39.694961 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:39.694913 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2509af5e-c021-4342-b2f3-7448be0ce972-original-pull-secret\") pod \"global-pull-secret-syncer-j62sj\" (UID: \"2509af5e-c021-4342-b2f3-7448be0ce972\") " pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:01:39.782476 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:39.782435 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs\") pod \"network-metrics-daemon-kpm26\" (UID: \"88ca28c8-725b-46f7-becc-ccd04207fba9\") " pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:01:39.782671 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:39.782508 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmlp\" (UniqueName: \"kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp\") pod \"network-check-target-gwkcf\" (UID: \"3eb2cd17-af75-4a86-bf49-2ff369e4860d\") " pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:01:39.785147 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:39.785124 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvmlp\" (UniqueName: \"kubernetes.io/projected/3eb2cd17-af75-4a86-bf49-2ff369e4860d-kube-api-access-qvmlp\") pod \"network-check-target-gwkcf\" (UID: \"3eb2cd17-af75-4a86-bf49-2ff369e4860d\") " pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:01:39.785717 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:39.785699 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 19:01:39.795421 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:39.795392 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88ca28c8-725b-46f7-becc-ccd04207fba9-metrics-certs\") pod \"network-metrics-daemon-kpm26\" (UID: \"88ca28c8-725b-46f7-becc-ccd04207fba9\") " pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:01:39.871710 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:39.871675 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sqhv9\"" Jul 10 19:01:39.875221 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:39.875204 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j62sj" Jul 10 19:01:39.879334 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:39.879315 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:01:39.884433 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:39.884417 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zb6nn\"" Jul 10 19:01:39.892125 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:39.892108 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kpm26" Jul 10 19:01:40.050436 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:40.050407 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j62sj"] Jul 10 19:01:40.054907 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:01:40.054869 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2509af5e_c021_4342_b2f3_7448be0ce972.slice/crio-124769e7e693230fb93556a7c397af76e8d8e61c20fad228e1c2f175027cafea WatchSource:0}: Error finding container 124769e7e693230fb93556a7c397af76e8d8e61c20fad228e1c2f175027cafea: Status 404 returned error can't find the container with id 124769e7e693230fb93556a7c397af76e8d8e61c20fad228e1c2f175027cafea Jul 10 19:01:40.069642 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:40.069537 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gwkcf"] Jul 10 19:01:40.072220 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:01:40.072195 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb2cd17_af75_4a86_bf49_2ff369e4860d.slice/crio-b03456868af1ae643bf6c0921195fe0f07eb4b88a8bb2d7f50d570dd41adfb09 WatchSource:0}: Error finding container b03456868af1ae643bf6c0921195fe0f07eb4b88a8bb2d7f50d570dd41adfb09: Status 404 returned error can't find the container with id b03456868af1ae643bf6c0921195fe0f07eb4b88a8bb2d7f50d570dd41adfb09 Jul 10 19:01:40.096875 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:40.096846 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kpm26"] Jul 10 19:01:40.100506 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:01:40.100483 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ca28c8_725b_46f7_becc_ccd04207fba9.slice/crio-b62e072d1aae2ae3ef768d154db759e421cf76dd7bb544ccb34631a9851a5852 WatchSource:0}: Error finding container b62e072d1aae2ae3ef768d154db759e421cf76dd7bb544ccb34631a9851a5852: Status 404 returned error can't find the container with id b62e072d1aae2ae3ef768d154db759e421cf76dd7bb544ccb34631a9851a5852 Jul 10 19:01:40.248228 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:40.248201 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hlcx4" Jul 10 19:01:40.286966 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:40.282979 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kpm26" event={"ID":"88ca28c8-725b-46f7-becc-ccd04207fba9","Type":"ContainerStarted","Data":"b62e072d1aae2ae3ef768d154db759e421cf76dd7bb544ccb34631a9851a5852"} Jul 10 19:01:40.286966 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:40.284762 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gwkcf" event={"ID":"3eb2cd17-af75-4a86-bf49-2ff369e4860d","Type":"ContainerStarted","Data":"0e142468e07c3c7c3ebebf8808610d3bab89075ba708b1da34e528671dd4de4d"} Jul 10 19:01:40.286966 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:40.284811 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gwkcf" event={"ID":"3eb2cd17-af75-4a86-bf49-2ff369e4860d","Type":"ContainerStarted","Data":"b03456868af1ae643bf6c0921195fe0f07eb4b88a8bb2d7f50d570dd41adfb09"} Jul 10 19:01:40.286966 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:40.284839 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:01:40.286966 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:40.286289 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j62sj" event={"ID":"2509af5e-c021-4342-b2f3-7448be0ce972","Type":"ContainerStarted","Data":"124769e7e693230fb93556a7c397af76e8d8e61c20fad228e1c2f175027cafea"} Jul 10 19:01:40.309286 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:40.309235 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gwkcf" podStartSLOduration=66.309219879 podStartE2EDuration="1m6.309219879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:40.308767045 +0000 UTC m=+66.909909182" watchObservedRunningTime="2026-07-10 19:01:40.309219879 +0000 UTC m=+66.910362014" Jul 10 19:01:40.325782 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:40.325756 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:40.406505 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:40.406471 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:42.296636 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:42.296590 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kpm26" event={"ID":"88ca28c8-725b-46f7-becc-ccd04207fba9","Type":"ContainerStarted","Data":"04c37748d7a4016f07bd326190435be5e0a437b444b1aa04b4b14e181ba1cbcb"} Jul 10 19:01:42.297108 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:42.296665 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kpm26" event={"ID":"88ca28c8-725b-46f7-becc-ccd04207fba9","Type":"ContainerStarted","Data":"b8bbc166482c663cbbb861a561eab5696553c9c2e19c15cb48723161d98a561f"} Jul 10 19:01:42.317642 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:42.317581 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kpm26" podStartSLOduration=66.80891458400001 podStartE2EDuration="1m8.317565666s" totalImagesPullingTime="1.508651082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:42.316378076 +0000 UTC m=+68.917520213" watchObservedRunningTime="2026-07-10 19:01:42.317565666 +0000 UTC m=+68.918707800" Jul 10 19:01:42.436654 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:42.436613 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:42.545478 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:42.545435 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:42.746834 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:42.746810 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:42.819285 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:42.819252 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:44.571475 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:44.571426 2561 patch_prober.go:28] interesting pod/image-registry-c68b6cf48-lxlf2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 19:01:44.571966 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:44.571521 2561 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" podUID="d95a6d04-03d4-4e92-b324-bca98f7ce082" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 19:01:44.913641 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:44.913606 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:45.154380 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.154284 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:45.233172 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.233140 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xs249"] Jul 10 19:01:45.235175 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.235159 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.238120 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.238084 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 19:01:45.238120 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.238106 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m7zfp\"" Jul 10 19:01:45.239343 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.239324 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 19:01:45.239450 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.239364 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 19:01:45.239450 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.239386 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 19:01:45.239564 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.239473 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 19:01:45.239625 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.239591 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 19:01:45.250503 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.250480 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:45.315017 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.314977 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j62sj" event={"ID":"2509af5e-c021-4342-b2f3-7448be0ce972","Type":"ContainerStarted","Data":"35550c23ff2250ed662a62b01407192cffd58e15ba07fe6cd0a52004f6684a18"} Jul 10 19:01:45.324008 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.323981 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-node-exporter-accelerators-collector-config\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.324008 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.324012 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqwln\" (UniqueName: \"kubernetes.io/projected/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-kube-api-access-cqwln\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.324215 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.324056 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-node-exporter-textfile\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.324215 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.324074 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-root\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.324215 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.324093 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-node-exporter-tls\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.324215 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.324131 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-node-exporter-wtmp\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.324215 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.324170 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-sys\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.324215 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.324193 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-metrics-client-ca\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.324506 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.324217 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.338974 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.338911 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j62sj" podStartSLOduration=66.611955909 podStartE2EDuration="1m11.338897159s" totalImagesPullingTime="4.72694125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:00:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:45.337583409 +0000 UTC m=+71.938725545" watchObservedRunningTime="2026-07-10 19:01:45.338897159 +0000 UTC m=+71.940039293" Jul 10 19:01:45.345620 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.345593 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:45.425567 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.425464 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-sys\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.425567 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.425509 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-metrics-client-ca\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.425567 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.425532 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.425834 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.425573 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-node-exporter-accelerators-collector-config\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.425834 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.425586 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-sys\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.425834 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.425602 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqwln\" (UniqueName: \"kubernetes.io/projected/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-kube-api-access-cqwln\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.425834 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.425677 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-node-exporter-textfile\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.425834 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.425708 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-root\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.425834 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.425743 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-node-exporter-tls\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.425834 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.425795 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-node-exporter-wtmp\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.426347 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.425970 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-root\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.426347 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.425987 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-node-exporter-wtmp\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.426347 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.426171 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-metrics-client-ca\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.426347 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.426269 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-node-exporter-textfile\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.426592 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.426486 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-node-exporter-accelerators-collector-config\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.428203 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.428181 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.428534 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.428510 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-node-exporter-tls\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.435133 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.435106 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqwln\" (UniqueName: \"kubernetes.io/projected/42e4fcd1-edab-49e1-9ac4-8462e1a43d2f-kube-api-access-cqwln\") pod \"node-exporter-xs249\" (UID: \"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f\") " pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.471911 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.471879 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:45.546899 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.546861 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xs249" Jul 10 19:01:45.556981 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:01:45.556950 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e4fcd1_edab_49e1_9ac4_8462e1a43d2f.slice/crio-7c57058ecac59f3897bd72fc3fd537ad51f5bdc2fc061d9559b329b7afc3ed4d WatchSource:0}: Error finding container 7c57058ecac59f3897bd72fc3fd537ad51f5bdc2fc061d9559b329b7afc3ed4d: Status 404 returned error can't find the container with id 7c57058ecac59f3897bd72fc3fd537ad51f5bdc2fc061d9559b329b7afc3ed4d Jul 10 19:01:45.574416 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.574387 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:45.671133 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.671101 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:45.771329 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:45.771297 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:46.216012 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:46.215917 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:46.320313 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:46.320280 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xs249" event={"ID":"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f","Type":"ContainerStarted","Data":"7c57058ecac59f3897bd72fc3fd537ad51f5bdc2fc061d9559b329b7afc3ed4d"} Jul 10 19:01:46.333841 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:46.333806 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:46.512449 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:46.512414 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:46.737482 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:46.737443 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:47.127630 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:47.127594 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:47.229355 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:47.229252 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:47.325747 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:47.325716 2561 generic.go:356] "Generic (PLEG): container finished" podID="42e4fcd1-edab-49e1-9ac4-8462e1a43d2f" containerID="af6e56cbf36e576c7eb364d50d966a0dc73cd347c79ed72f58f6d0c9a314b26e" exitCode=0 Jul 10 19:01:47.325905 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:47.325803 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xs249" event={"ID":"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f","Type":"ContainerDied","Data":"af6e56cbf36e576c7eb364d50d966a0dc73cd347c79ed72f58f6d0c9a314b26e"} Jul 10 19:01:47.897017 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:47.896982 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:48.002052 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.002021 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:48.203856 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.203764 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:48.205689 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.205666 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-57769bbb47-876fw"] Jul 10 19:01:48.208838 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.208819 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.212011 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.211987 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 10 19:01:48.212157 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.212134 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 10 19:01:48.213353 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.213185 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-9w6tf\"" Jul 10 19:01:48.213353 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.213201 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 10 19:01:48.213353 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.213219 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 10 19:01:48.213353 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.213249 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4821t912a33eo\"" Jul 10 19:01:48.213608 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.213400 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 10 19:01:48.222823 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.222802 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57769bbb47-876fw"] Jul 10 19:01:48.309832 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.309795 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:48.332666 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.332629 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xs249" event={"ID":"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f","Type":"ContainerStarted","Data":"74ab3926e74d37f1fd91825cc1da7a73ee6237ac77cd13b84f6cd224d2c037cd"} Jul 10 19:01:48.332829 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.332692 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xs249" event={"ID":"42e4fcd1-edab-49e1-9ac4-8462e1a43d2f","Type":"ContainerStarted","Data":"d7eb51abec7867da1e2fed8f03ccecfc3a9a3781cec0e2b94025830f49419286"} Jul 10 19:01:48.348620 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.348585 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.348732 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.348634 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-secret-thanos-querier-tls\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.348732 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.348664 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.348842 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.348725 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.348954 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.348764 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-metrics-client-ca\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.349219 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.349194 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szjgz\" (UniqueName: \"kubernetes.io/projected/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-kube-api-access-szjgz\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.349289 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.349277 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-secret-grpc-tls\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.352956 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.349331 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.450599 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.450556 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.450799 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.450623 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.450799 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.450658 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-secret-thanos-querier-tls\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.450799 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.450687 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.450799 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.450720 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.450799 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.450767 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-metrics-client-ca\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.450799 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.450799 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szjgz\" (UniqueName: \"kubernetes.io/projected/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-kube-api-access-szjgz\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.451139 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.450859 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-secret-grpc-tls\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.451654 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.451606 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-metrics-client-ca\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.454176 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.454114 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.454459 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.454436 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.454536 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.454514 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.454630 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.454606 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.454742 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.454721 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-secret-thanos-querier-tls\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.455093 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.455076 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-secret-grpc-tls\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.459926 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.459902 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szjgz\" (UniqueName: \"kubernetes.io/projected/6f3f76d4-602c-4c81-9a2a-c8679e91ac00-kube-api-access-szjgz\") pod \"thanos-querier-57769bbb47-876fw\" (UID: \"6f3f76d4-602c-4c81-9a2a-c8679e91ac00\") " pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.517799 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.517765 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:48.664082 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.664018 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xs249" podStartSLOduration=2.221287872 podStartE2EDuration="3.663994875s" totalImagesPullingTime="1.371162219s" totalInitContainerRuntime="71.544784ms" isStatefulPod=true podCreationTimestamp="2026-07-10 19:01:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:48.355183435 +0000 UTC m=+74.956325581" watchObservedRunningTime="2026-07-10 19:01:48.663994875 +0000 UTC m=+75.265137011" Jul 10 19:01:48.665093 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.665074 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57769bbb47-876fw"] Jul 10 19:01:48.667640 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:01:48.667613 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f3f76d4_602c_4c81_9a2a_c8679e91ac00.slice/crio-b37735aadee3bf051dbdec3cafddb7494c295127a896ee0a5667ac7fd7abf195 WatchSource:0}: Error finding container b37735aadee3bf051dbdec3cafddb7494c295127a896ee0a5667ac7fd7abf195: Status 404 returned error can't find the container with id b37735aadee3bf051dbdec3cafddb7494c295127a896ee0a5667ac7fd7abf195 Jul 10 19:01:48.714605 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.714516 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:48.859172 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:48.859135 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:49.337901 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:49.337862 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" event={"ID":"6f3f76d4-602c-4c81-9a2a-c8679e91ac00","Type":"ContainerStarted","Data":"b37735aadee3bf051dbdec3cafddb7494c295127a896ee0a5667ac7fd7abf195"} Jul 10 19:01:49.499713 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:49.499679 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:49.602565 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:49.602478 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:49.770355 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:49.770322 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:49.968823 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:49.968738 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:50.093087 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:50.093050 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:50.649675 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:50.649639 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:51.347414 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:51.347386 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" event={"ID":"6f3f76d4-602c-4c81-9a2a-c8679e91ac00","Type":"ContainerStarted","Data":"05285ba40915568cab486f49fca2d11107c4b77752c2e7b8a207d07b762f1cbb"} Jul 10 19:01:51.529986 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:51.529955 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:52.356691 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:52.356665 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" event={"ID":"6f3f76d4-602c-4c81-9a2a-c8679e91ac00","Type":"ContainerStarted","Data":"a6eeb889412c72cca3a456794965d6251d7c9a58ffdfed9ab6f5978761b570a3"} Jul 10 19:01:52.357069 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:52.356729 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" event={"ID":"6f3f76d4-602c-4c81-9a2a-c8679e91ac00","Type":"ContainerStarted","Data":"f21fdd97a8b09b4b2831d5a5c5640edef13b0f316ce436fdbbae9bc5af2430f5"} Jul 10 19:01:52.357069 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:52.356758 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" event={"ID":"6f3f76d4-602c-4c81-9a2a-c8679e91ac00","Type":"ContainerStarted","Data":"84ccc1281462c8d7e774c2ff19610e9025ae81e4c953c27d2af546d1a44e1765"} Jul 10 19:01:53.364059 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:53.364015 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" event={"ID":"6f3f76d4-602c-4c81-9a2a-c8679e91ac00","Type":"ContainerStarted","Data":"19eb507daab8223d0493c954943e05b7045c24b776cc0fbf1eec3de5f0db1aa4"} Jul 10 19:01:53.364517 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:53.364160 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" event={"ID":"6f3f76d4-602c-4c81-9a2a-c8679e91ac00","Type":"ContainerStarted","Data":"0328ed9b048818cac28ca5ac37cb8ef1355660177213aaca6de500dcc6216656"} Jul 10 19:01:53.365554 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:53.365529 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:01:53.381388 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:53.381345 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:53.396107 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:53.396047 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" podStartSLOduration=2.006621666 podStartE2EDuration="5.396031781s" totalImagesPullingTime="3.389410115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:01:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:01:53.394673626 +0000 UTC m=+79.995815760" watchObservedRunningTime="2026-07-10 19:01:53.396031781 +0000 UTC m=+79.997173916" Jul 10 19:01:53.526174 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:53.526142 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:54.569196 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:54.569167 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:55.175866 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:55.175835 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:55.294902 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:55.294862 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:56.232234 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:56.232196 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:56.335041 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:56.335003 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:57.442589 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:57.442551 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:57.548057 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:57.548019 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:58.581649 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:58.581613 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:58.680436 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:58.680402 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:01:59.583587 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.583549 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" podUID="d95a6d04-03d4-4e92-b324-bca98f7ce082" containerName="registry" containerID="cri-o://880bb2783072bd5b003d28a4b6ba5c1fa0a59dd7e6911783070b5da0b58bf35e" gracePeriod=30 Jul 10 19:01:59.838889 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.838825 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:01:59.951355 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.951315 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls\") pod \"d95a6d04-03d4-4e92-b324-bca98f7ce082\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " Jul 10 19:01:59.951505 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.951379 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-certificates\") pod \"d95a6d04-03d4-4e92-b324-bca98f7ce082\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " Jul 10 19:01:59.951505 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.951405 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtsjk\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-kube-api-access-vtsjk\") pod \"d95a6d04-03d4-4e92-b324-bca98f7ce082\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " Jul 10 19:01:59.951505 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.951437 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d95a6d04-03d4-4e92-b324-bca98f7ce082-installation-pull-secrets\") pod \"d95a6d04-03d4-4e92-b324-bca98f7ce082\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " Jul 10 19:01:59.951505 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.951473 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-bound-sa-token\") pod \"d95a6d04-03d4-4e92-b324-bca98f7ce082\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " Jul 10 19:01:59.951693 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.951559 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d95a6d04-03d4-4e92-b324-bca98f7ce082-trusted-ca\") pod \"d95a6d04-03d4-4e92-b324-bca98f7ce082\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " Jul 10 19:01:59.951693 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.951625 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d95a6d04-03d4-4e92-b324-bca98f7ce082-image-registry-private-configuration\") pod \"d95a6d04-03d4-4e92-b324-bca98f7ce082\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " Jul 10 19:01:59.951786 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.951768 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d95a6d04-03d4-4e92-b324-bca98f7ce082-ca-trust-extracted\") pod \"d95a6d04-03d4-4e92-b324-bca98f7ce082\" (UID: \"d95a6d04-03d4-4e92-b324-bca98f7ce082\") " Jul 10 19:01:59.952134 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.951972 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d95a6d04-03d4-4e92-b324-bca98f7ce082" (UID: "d95a6d04-03d4-4e92-b324-bca98f7ce082"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:01:59.952429 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.952234 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95a6d04-03d4-4e92-b324-bca98f7ce082-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d95a6d04-03d4-4e92-b324-bca98f7ce082" (UID: "d95a6d04-03d4-4e92-b324-bca98f7ce082"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 19:01:59.952626 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.952599 2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-certificates\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:01:59.952745 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.952631 2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d95a6d04-03d4-4e92-b324-bca98f7ce082-trusted-ca\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:01:59.954230 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.954180 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95a6d04-03d4-4e92-b324-bca98f7ce082-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d95a6d04-03d4-4e92-b324-bca98f7ce082" (UID: "d95a6d04-03d4-4e92-b324-bca98f7ce082"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:01:59.954324 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.954288 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d95a6d04-03d4-4e92-b324-bca98f7ce082" (UID: "d95a6d04-03d4-4e92-b324-bca98f7ce082"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:01:59.954382 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.954357 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95a6d04-03d4-4e92-b324-bca98f7ce082-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d95a6d04-03d4-4e92-b324-bca98f7ce082" (UID: "d95a6d04-03d4-4e92-b324-bca98f7ce082"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:01:59.954427 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.954379 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d95a6d04-03d4-4e92-b324-bca98f7ce082" (UID: "d95a6d04-03d4-4e92-b324-bca98f7ce082"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:01:59.954460 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.954442 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-kube-api-access-vtsjk" (OuterVolumeSpecName: "kube-api-access-vtsjk") pod "d95a6d04-03d4-4e92-b324-bca98f7ce082" (UID: "d95a6d04-03d4-4e92-b324-bca98f7ce082"). InnerVolumeSpecName "kube-api-access-vtsjk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:01:59.961645 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:01:59.961620 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d95a6d04-03d4-4e92-b324-bca98f7ce082-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d95a6d04-03d4-4e92-b324-bca98f7ce082" (UID: "d95a6d04-03d4-4e92-b324-bca98f7ce082"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:02:00.053539 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:00.053496 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtsjk\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-kube-api-access-vtsjk\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:02:00.053539 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:00.053531 2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d95a6d04-03d4-4e92-b324-bca98f7ce082-installation-pull-secrets\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:02:00.053539 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:00.053541 2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-bound-sa-token\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:02:00.053766 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:00.053550 2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d95a6d04-03d4-4e92-b324-bca98f7ce082-image-registry-private-configuration\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:02:00.053766 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:00.053559 2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d95a6d04-03d4-4e92-b324-bca98f7ce082-ca-trust-extracted\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:02:00.053766 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:00.053567 2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d95a6d04-03d4-4e92-b324-bca98f7ce082-registry-tls\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:02:00.377826 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:00.377801 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-57769bbb47-876fw" Jul 10 19:02:00.389797 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:00.389767 2561 generic.go:356] "Generic (PLEG): container finished" podID="d95a6d04-03d4-4e92-b324-bca98f7ce082" containerID="880bb2783072bd5b003d28a4b6ba5c1fa0a59dd7e6911783070b5da0b58bf35e" exitCode=0 Jul 10 19:02:00.389976 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:00.389812 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" Jul 10 19:02:00.389976 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:00.389842 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" event={"ID":"d95a6d04-03d4-4e92-b324-bca98f7ce082","Type":"ContainerDied","Data":"880bb2783072bd5b003d28a4b6ba5c1fa0a59dd7e6911783070b5da0b58bf35e"} Jul 10 19:02:00.389976 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:00.389888 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c68b6cf48-lxlf2" event={"ID":"d95a6d04-03d4-4e92-b324-bca98f7ce082","Type":"ContainerDied","Data":"f6a92bf610dee666b352f35066c6ecd56e93395896aae56c57fe9a1c35f15ff1"} Jul 10 19:02:00.389976 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:00.389909 2561 scope.go:117] "RemoveContainer" containerID="880bb2783072bd5b003d28a4b6ba5c1fa0a59dd7e6911783070b5da0b58bf35e" Jul 10 19:02:00.398421 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:00.398398 2561 scope.go:117] "RemoveContainer" containerID="880bb2783072bd5b003d28a4b6ba5c1fa0a59dd7e6911783070b5da0b58bf35e" Jul 10 19:02:00.398718 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:02:00.398694 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"880bb2783072bd5b003d28a4b6ba5c1fa0a59dd7e6911783070b5da0b58bf35e\": container with ID starting with 880bb2783072bd5b003d28a4b6ba5c1fa0a59dd7e6911783070b5da0b58bf35e not found: ID does not exist" containerID="880bb2783072bd5b003d28a4b6ba5c1fa0a59dd7e6911783070b5da0b58bf35e" Jul 10 19:02:00.398783 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:00.398727 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"880bb2783072bd5b003d28a4b6ba5c1fa0a59dd7e6911783070b5da0b58bf35e"} err="failed to get container status \"880bb2783072bd5b003d28a4b6ba5c1fa0a59dd7e6911783070b5da0b58bf35e\": rpc error: code = NotFound desc = could not find container \"880bb2783072bd5b003d28a4b6ba5c1fa0a59dd7e6911783070b5da0b58bf35e\": container with ID starting with 880bb2783072bd5b003d28a4b6ba5c1fa0a59dd7e6911783070b5da0b58bf35e not found: ID does not exist" Jul 10 19:02:00.435961 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:00.435911 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-c68b6cf48-lxlf2"] Jul 10 19:02:00.446248 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:00.446220 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-c68b6cf48-lxlf2"] Jul 10 19:02:01.698194 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:01.698160 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:01.801449 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:01.801411 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:01.962034 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:01.961958 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d95a6d04-03d4-4e92-b324-bca98f7ce082" path="/var/lib/kubelet/pods/d95a6d04-03d4-4e92-b324-bca98f7ce082/volumes" Jul 10 19:02:02.282050 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:02.282014 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:02.395696 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:02.395655 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:03.465343 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:03.465312 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:03.609990 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:03.609960 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:10.256075 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:10.256035 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:10.367633 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:10.367596 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:11.294172 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:11.294143 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gwkcf" Jul 10 19:02:14.866552 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:14.866520 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:14.965772 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:14.965734 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:15.034814 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:15.034766 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:17.541489 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:17.541450 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:17.644664 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:17.644630 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:17.746858 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:17.746822 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:21.796730 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:21.796685 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:21.875167 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:21.875135 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:25.258023 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:25.257989 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:25.360993 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:25.360964 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:31.954256 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:31.954209 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:32.036787 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:32.036741 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:32.104098 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:32.104063 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:02:58.179327 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:02:58.179292 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:01.322650 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:01.322608 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:01.438435 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:01.438388 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:02.497167 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:02.497135 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:02.597642 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:02.597571 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:02.699291 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:02.699256 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:03.220984 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:03.220954 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:03.289118 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:03.289079 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:09.900414 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:09.900377 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:13.321570 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:13.321536 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:13.422698 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:13.422666 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:13.840301 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:13.840253 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:13.939333 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:13.939298 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:14.005807 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:14.005768 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:25.258634 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:25.258547 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:25.428083 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:25.428048 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:44.913025 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:44.912986 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:45.009695 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:45.009661 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:45.996488 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:45.996462 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4"] Jul 10 19:03:45.996846 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:45.996706 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d95a6d04-03d4-4e92-b324-bca98f7ce082" containerName="registry" Jul 10 19:03:45.996846 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:45.996716 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d95a6d04-03d4-4e92-b324-bca98f7ce082" containerName="registry" Jul 10 19:03:45.996846 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:45.996763 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d95a6d04-03d4-4e92-b324-bca98f7ce082" containerName="registry" Jul 10 19:03:45.999731 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:45.999715 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4" Jul 10 19:03:46.002257 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:46.002236 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 19:03:46.003294 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:46.003276 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hkrss\"" Jul 10 19:03:46.003387 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:46.003316 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 19:03:46.008789 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:46.008762 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4"] Jul 10 19:03:46.117200 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:46.117163 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e96263ec-7d87-4d41-b156-5c32e6fbe30d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4\" (UID: \"e96263ec-7d87-4d41-b156-5c32e6fbe30d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4" Jul 10 19:03:46.117376 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:46.117228 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hxb6\" (UniqueName: \"kubernetes.io/projected/e96263ec-7d87-4d41-b156-5c32e6fbe30d-kube-api-access-6hxb6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4\" (UID: \"e96263ec-7d87-4d41-b156-5c32e6fbe30d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4" Jul 10 19:03:46.117376 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:46.117295 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e96263ec-7d87-4d41-b156-5c32e6fbe30d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4\" (UID: \"e96263ec-7d87-4d41-b156-5c32e6fbe30d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4" Jul 10 19:03:46.217964 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:46.217913 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e96263ec-7d87-4d41-b156-5c32e6fbe30d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4\" (UID: \"e96263ec-7d87-4d41-b156-5c32e6fbe30d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4" Jul 10 19:03:46.218085 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:46.217977 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hxb6\" (UniqueName: \"kubernetes.io/projected/e96263ec-7d87-4d41-b156-5c32e6fbe30d-kube-api-access-6hxb6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4\" (UID: \"e96263ec-7d87-4d41-b156-5c32e6fbe30d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4" Jul 10 19:03:46.218085 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:46.218016 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e96263ec-7d87-4d41-b156-5c32e6fbe30d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4\" (UID: \"e96263ec-7d87-4d41-b156-5c32e6fbe30d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4" Jul 10 19:03:46.218286 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:46.218264 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e96263ec-7d87-4d41-b156-5c32e6fbe30d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4\" (UID: \"e96263ec-7d87-4d41-b156-5c32e6fbe30d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4" Jul 10 19:03:46.218342 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:46.218323 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e96263ec-7d87-4d41-b156-5c32e6fbe30d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4\" (UID: \"e96263ec-7d87-4d41-b156-5c32e6fbe30d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4" Jul 10 19:03:46.228135 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:46.228102 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hxb6\" (UniqueName: \"kubernetes.io/projected/e96263ec-7d87-4d41-b156-5c32e6fbe30d-kube-api-access-6hxb6\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4\" (UID: \"e96263ec-7d87-4d41-b156-5c32e6fbe30d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4" Jul 10 19:03:46.308521 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:46.308419 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4" Jul 10 19:03:46.436873 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:46.436797 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4"] Jul 10 19:03:46.439497 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:03:46.439465 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96263ec_7d87_4d41_b156_5c32e6fbe30d.slice/crio-98d833e90042503525d1c8050febb837101ac84f515121806b7333c3c74e2d1d WatchSource:0}: Error finding container 98d833e90042503525d1c8050febb837101ac84f515121806b7333c3c74e2d1d: Status 404 returned error can't find the container with id 98d833e90042503525d1c8050febb837101ac84f515121806b7333c3c74e2d1d Jul 10 19:03:46.755112 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:46.755076 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4" event={"ID":"e96263ec-7d87-4d41-b156-5c32e6fbe30d","Type":"ContainerStarted","Data":"98d833e90042503525d1c8050febb837101ac84f515121806b7333c3c74e2d1d"} Jul 10 19:03:47.902795 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:47.902746 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:47.981598 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:47.981557 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:03:52.779866 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:52.779831 2561 generic.go:356] "Generic (PLEG): container finished" podID="e96263ec-7d87-4d41-b156-5c32e6fbe30d" containerID="9d0fb28813e512ed6f14f93d973371f6bb0d9a6c61f7ed76e017c54e914b1f15" exitCode=0 Jul 10 19:03:52.780274 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:52.779876 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4" event={"ID":"e96263ec-7d87-4d41-b156-5c32e6fbe30d","Type":"ContainerDied","Data":"9d0fb28813e512ed6f14f93d973371f6bb0d9a6c61f7ed76e017c54e914b1f15"} Jul 10 19:03:55.792198 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:55.792160 2561 generic.go:356] "Generic (PLEG): container finished" podID="e96263ec-7d87-4d41-b156-5c32e6fbe30d" containerID="7ff29d3e0401367b6511a311549f5f47269fa1cec5b04f9335f2609ea727b4b2" exitCode=0 Jul 10 19:03:55.792595 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:03:55.792216 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4" event={"ID":"e96263ec-7d87-4d41-b156-5c32e6fbe30d","Type":"ContainerDied","Data":"7ff29d3e0401367b6511a311549f5f47269fa1cec5b04f9335f2609ea727b4b2"} Jul 10 19:04:02.820543 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:02.820508 2561 generic.go:356] "Generic (PLEG): container finished" podID="e96263ec-7d87-4d41-b156-5c32e6fbe30d" containerID="ad52009af11540a82681b62c9fc7167030ad15baf42944ae61744bffe4849973" exitCode=0 Jul 10 19:04:02.821003 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:02.820600 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4" event={"ID":"e96263ec-7d87-4d41-b156-5c32e6fbe30d","Type":"ContainerDied","Data":"ad52009af11540a82681b62c9fc7167030ad15baf42944ae61744bffe4849973"} Jul 10 19:04:03.946335 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:03.946311 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4" Jul 10 19:04:03.957407 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:03.957380 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hxb6\" (UniqueName: \"kubernetes.io/projected/e96263ec-7d87-4d41-b156-5c32e6fbe30d-kube-api-access-6hxb6\") pod \"e96263ec-7d87-4d41-b156-5c32e6fbe30d\" (UID: \"e96263ec-7d87-4d41-b156-5c32e6fbe30d\") " Jul 10 19:04:03.957533 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:03.957424 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e96263ec-7d87-4d41-b156-5c32e6fbe30d-bundle\") pod \"e96263ec-7d87-4d41-b156-5c32e6fbe30d\" (UID: \"e96263ec-7d87-4d41-b156-5c32e6fbe30d\") " Jul 10 19:04:03.957598 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:03.957585 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e96263ec-7d87-4d41-b156-5c32e6fbe30d-util\") pod \"e96263ec-7d87-4d41-b156-5c32e6fbe30d\" (UID: \"e96263ec-7d87-4d41-b156-5c32e6fbe30d\") " Jul 10 19:04:03.958139 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:03.958113 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96263ec-7d87-4d41-b156-5c32e6fbe30d-bundle" (OuterVolumeSpecName: "bundle") pod "e96263ec-7d87-4d41-b156-5c32e6fbe30d" (UID: "e96263ec-7d87-4d41-b156-5c32e6fbe30d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:04:03.959656 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:03.959631 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96263ec-7d87-4d41-b156-5c32e6fbe30d-kube-api-access-6hxb6" (OuterVolumeSpecName: "kube-api-access-6hxb6") pod "e96263ec-7d87-4d41-b156-5c32e6fbe30d" (UID: "e96263ec-7d87-4d41-b156-5c32e6fbe30d"). InnerVolumeSpecName "kube-api-access-6hxb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:04:03.962925 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:03.962902 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e96263ec-7d87-4d41-b156-5c32e6fbe30d-util" (OuterVolumeSpecName: "util") pod "e96263ec-7d87-4d41-b156-5c32e6fbe30d" (UID: "e96263ec-7d87-4d41-b156-5c32e6fbe30d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:04:04.058241 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:04.058200 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hxb6\" (UniqueName: \"kubernetes.io/projected/e96263ec-7d87-4d41-b156-5c32e6fbe30d-kube-api-access-6hxb6\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:04:04.058241 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:04.058233 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e96263ec-7d87-4d41-b156-5c32e6fbe30d-bundle\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:04:04.058241 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:04.058243 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e96263ec-7d87-4d41-b156-5c32e6fbe30d-util\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:04:04.829236 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:04.829199 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4" event={"ID":"e96263ec-7d87-4d41-b156-5c32e6fbe30d","Type":"ContainerDied","Data":"98d833e90042503525d1c8050febb837101ac84f515121806b7333c3c74e2d1d"} Jul 10 19:04:04.829403 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:04.829248 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540q2gf4" Jul 10 19:04:04.829403 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:04.829255 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98d833e90042503525d1c8050febb837101ac84f515121806b7333c3c74e2d1d" Jul 10 19:04:06.943874 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:06.943836 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:07.750711 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:07.750680 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:08.900838 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:08.900805 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kw2tk"] Jul 10 19:04:08.901205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:08.901072 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e96263ec-7d87-4d41-b156-5c32e6fbe30d" containerName="extract" Jul 10 19:04:08.901205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:08.901083 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e96263ec-7d87-4d41-b156-5c32e6fbe30d" containerName="extract" Jul 10 19:04:08.901205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:08.901100 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e96263ec-7d87-4d41-b156-5c32e6fbe30d" containerName="pull" Jul 10 19:04:08.901205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:08.901105 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e96263ec-7d87-4d41-b156-5c32e6fbe30d" containerName="pull" Jul 10 19:04:08.901205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:08.901113 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e96263ec-7d87-4d41-b156-5c32e6fbe30d" containerName="util" Jul 10 19:04:08.901205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:08.901119 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="e96263ec-7d87-4d41-b156-5c32e6fbe30d" containerName="util" Jul 10 19:04:08.901205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:08.901162 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="e96263ec-7d87-4d41-b156-5c32e6fbe30d" containerName="extract" Jul 10 19:04:08.937334 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:08.937297 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kw2tk"] Jul 10 19:04:08.937469 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:08.937436 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kw2tk" Jul 10 19:04:08.940212 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:08.940191 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 10 19:04:08.940355 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:08.940241 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-lxt5l\"" Jul 10 19:04:08.940355 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:08.940304 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 10 19:04:08.985954 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:08.985905 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32027b9b-018e-4949-b9e3-ecad6c788c46-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kw2tk\" (UID: \"32027b9b-018e-4949-b9e3-ecad6c788c46\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kw2tk" Jul 10 19:04:08.986115 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:08.985980 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n9vd\" (UniqueName: \"kubernetes.io/projected/32027b9b-018e-4949-b9e3-ecad6c788c46-kube-api-access-4n9vd\") pod \"cert-manager-operator-controller-manager-55c444bf86-kw2tk\" (UID: \"32027b9b-018e-4949-b9e3-ecad6c788c46\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kw2tk" Jul 10 19:04:09.086878 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:09.086834 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n9vd\" (UniqueName: \"kubernetes.io/projected/32027b9b-018e-4949-b9e3-ecad6c788c46-kube-api-access-4n9vd\") pod \"cert-manager-operator-controller-manager-55c444bf86-kw2tk\" (UID: \"32027b9b-018e-4949-b9e3-ecad6c788c46\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kw2tk" Jul 10 19:04:09.087086 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:09.086923 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32027b9b-018e-4949-b9e3-ecad6c788c46-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kw2tk\" (UID: \"32027b9b-018e-4949-b9e3-ecad6c788c46\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kw2tk" Jul 10 19:04:09.087301 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:09.087281 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32027b9b-018e-4949-b9e3-ecad6c788c46-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-kw2tk\" (UID: \"32027b9b-018e-4949-b9e3-ecad6c788c46\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kw2tk" Jul 10 19:04:09.095474 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:09.095440 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n9vd\" (UniqueName: \"kubernetes.io/projected/32027b9b-018e-4949-b9e3-ecad6c788c46-kube-api-access-4n9vd\") pod \"cert-manager-operator-controller-manager-55c444bf86-kw2tk\" (UID: \"32027b9b-018e-4949-b9e3-ecad6c788c46\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kw2tk" Jul 10 19:04:09.246423 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:09.246395 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kw2tk" Jul 10 19:04:09.383205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:09.383174 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kw2tk"] Jul 10 19:04:09.387038 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:04:09.387011 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32027b9b_018e_4949_b9e3_ecad6c788c46.slice/crio-aaadfc51ab7af67b66ca2471738efe5cc7deea3f043bbd5a7410a91796679326 WatchSource:0}: Error finding container aaadfc51ab7af67b66ca2471738efe5cc7deea3f043bbd5a7410a91796679326: Status 404 returned error can't find the container with id aaadfc51ab7af67b66ca2471738efe5cc7deea3f043bbd5a7410a91796679326 Jul 10 19:04:09.848958 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:09.848902 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kw2tk" event={"ID":"32027b9b-018e-4949-b9e3-ecad6c788c46","Type":"ContainerStarted","Data":"aaadfc51ab7af67b66ca2471738efe5cc7deea3f043bbd5a7410a91796679326"} Jul 10 19:04:10.577323 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:10.577287 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:10.678326 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:10.678284 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:11.799145 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:11.799109 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:11.859814 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:11.859778 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kw2tk" event={"ID":"32027b9b-018e-4949-b9e3-ecad6c788c46","Type":"ContainerStarted","Data":"0d55f83c25ce27d2b49a6d20f18d927853c9bb0008613c0150d66f9c44381f94"} Jul 10 19:04:11.873202 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:11.873174 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:11.882021 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:11.881980 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-kw2tk" podStartSLOduration=1.839092774 podStartE2EDuration="3.881966575s" totalImagesPullingTime="2.042873801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:04:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:04:11.88084937 +0000 UTC m=+218.481991506" watchObservedRunningTime="2026-07-10 19:04:11.881966575 +0000 UTC m=+218.483108709" Jul 10 19:04:12.245650 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:12.245617 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:12.364533 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:12.364495 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:12.613136 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:12.611172 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8t67g"] Jul 10 19:04:12.631022 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:12.630990 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8t67g" Jul 10 19:04:12.634815 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:12.634786 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 19:04:12.635010 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:12.634786 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 19:04:12.635010 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:12.634786 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-kj7nn\"" Jul 10 19:04:12.635640 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:12.635613 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8t67g"] Jul 10 19:04:12.709910 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:12.709875 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78ea5c71-2618-4053-8797-4ca3ad24778e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8t67g\" (UID: \"78ea5c71-2618-4053-8797-4ca3ad24778e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8t67g" Jul 10 19:04:12.709910 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:12.709911 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg9k8\" (UniqueName: \"kubernetes.io/projected/78ea5c71-2618-4053-8797-4ca3ad24778e-kube-api-access-qg9k8\") pod \"cert-manager-webhook-6486ffd997-8t67g\" (UID: \"78ea5c71-2618-4053-8797-4ca3ad24778e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8t67g" Jul 10 19:04:12.810537 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:12.810501 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg9k8\" (UniqueName: \"kubernetes.io/projected/78ea5c71-2618-4053-8797-4ca3ad24778e-kube-api-access-qg9k8\") pod \"cert-manager-webhook-6486ffd997-8t67g\" (UID: \"78ea5c71-2618-4053-8797-4ca3ad24778e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8t67g" Jul 10 19:04:12.810898 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:12.810570 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78ea5c71-2618-4053-8797-4ca3ad24778e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8t67g\" (UID: \"78ea5c71-2618-4053-8797-4ca3ad24778e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8t67g" Jul 10 19:04:12.820188 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:12.820152 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/78ea5c71-2618-4053-8797-4ca3ad24778e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8t67g\" (UID: \"78ea5c71-2618-4053-8797-4ca3ad24778e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8t67g" Jul 10 19:04:12.820312 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:12.820219 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg9k8\" (UniqueName: \"kubernetes.io/projected/78ea5c71-2618-4053-8797-4ca3ad24778e-kube-api-access-qg9k8\") pod \"cert-manager-webhook-6486ffd997-8t67g\" (UID: \"78ea5c71-2618-4053-8797-4ca3ad24778e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8t67g" Jul 10 19:04:12.941685 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:12.941592 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8t67g" Jul 10 19:04:13.087088 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.087055 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8t67g"] Jul 10 19:04:13.090475 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:04:13.090448 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78ea5c71_2618_4053_8797_4ca3ad24778e.slice/crio-fcea51654ec568974870c8859586376ef3320dd9869b51785292836575765613 WatchSource:0}: Error finding container fcea51654ec568974870c8859586376ef3320dd9869b51785292836575765613: Status 404 returned error can't find the container with id fcea51654ec568974870c8859586376ef3320dd9869b51785292836575765613 Jul 10 19:04:13.240755 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.240672 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c"] Jul 10 19:04:13.272666 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.272633 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c"] Jul 10 19:04:13.272829 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.272743 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c" Jul 10 19:04:13.275829 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.275801 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 19:04:13.275960 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.275843 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 19:04:13.276038 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.275992 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hkrss\"" Jul 10 19:04:13.315349 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.315313 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bss6w\" (UniqueName: \"kubernetes.io/projected/90fa8380-0306-4dee-91d7-77fc4a573610-kube-api-access-bss6w\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c\" (UID: \"90fa8380-0306-4dee-91d7-77fc4a573610\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c" Jul 10 19:04:13.315512 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.315359 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90fa8380-0306-4dee-91d7-77fc4a573610-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c\" (UID: \"90fa8380-0306-4dee-91d7-77fc4a573610\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c" Jul 10 19:04:13.315512 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.315414 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90fa8380-0306-4dee-91d7-77fc4a573610-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c\" (UID: \"90fa8380-0306-4dee-91d7-77fc4a573610\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c" Jul 10 19:04:13.415747 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.415713 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bss6w\" (UniqueName: \"kubernetes.io/projected/90fa8380-0306-4dee-91d7-77fc4a573610-kube-api-access-bss6w\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c\" (UID: \"90fa8380-0306-4dee-91d7-77fc4a573610\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c" Jul 10 19:04:13.415747 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.415752 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90fa8380-0306-4dee-91d7-77fc4a573610-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c\" (UID: \"90fa8380-0306-4dee-91d7-77fc4a573610\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c" Jul 10 19:04:13.416038 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.415779 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90fa8380-0306-4dee-91d7-77fc4a573610-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c\" (UID: \"90fa8380-0306-4dee-91d7-77fc4a573610\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c" Jul 10 19:04:13.416157 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.416138 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90fa8380-0306-4dee-91d7-77fc4a573610-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c\" (UID: \"90fa8380-0306-4dee-91d7-77fc4a573610\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c" Jul 10 19:04:13.416215 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.416161 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90fa8380-0306-4dee-91d7-77fc4a573610-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c\" (UID: \"90fa8380-0306-4dee-91d7-77fc4a573610\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c" Jul 10 19:04:13.425655 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.425625 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bss6w\" (UniqueName: \"kubernetes.io/projected/90fa8380-0306-4dee-91d7-77fc4a573610-kube-api-access-bss6w\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c\" (UID: \"90fa8380-0306-4dee-91d7-77fc4a573610\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c" Jul 10 19:04:13.582399 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.582363 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c" Jul 10 19:04:13.734340 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.734297 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c"] Jul 10 19:04:13.737817 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:04:13.737791 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fa8380_0306_4dee_91d7_77fc4a573610.slice/crio-86427ed72d392b1dbadb397264250a8ba45042ad33a8617bf3f5a006150a105d WatchSource:0}: Error finding container 86427ed72d392b1dbadb397264250a8ba45042ad33a8617bf3f5a006150a105d: Status 404 returned error can't find the container with id 86427ed72d392b1dbadb397264250a8ba45042ad33a8617bf3f5a006150a105d Jul 10 19:04:13.870982 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.870919 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c" event={"ID":"90fa8380-0306-4dee-91d7-77fc4a573610","Type":"ContainerStarted","Data":"ff3ee17dce69c9a0266ebc3a61343f7e9ce38966b46111fd847ee1bf2ae51477"} Jul 10 19:04:13.871420 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.871003 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c" event={"ID":"90fa8380-0306-4dee-91d7-77fc4a573610","Type":"ContainerStarted","Data":"86427ed72d392b1dbadb397264250a8ba45042ad33a8617bf3f5a006150a105d"} Jul 10 19:04:13.873003 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:13.872973 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8t67g" event={"ID":"78ea5c71-2618-4053-8797-4ca3ad24778e","Type":"ContainerStarted","Data":"fcea51654ec568974870c8859586376ef3320dd9869b51785292836575765613"} Jul 10 19:04:14.634415 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:14.634380 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:14.837367 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:14.837328 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:14.879715 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:14.879680 2561 generic.go:356] "Generic (PLEG): container finished" podID="90fa8380-0306-4dee-91d7-77fc4a573610" containerID="ff3ee17dce69c9a0266ebc3a61343f7e9ce38966b46111fd847ee1bf2ae51477" exitCode=0 Jul 10 19:04:14.880211 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:14.879788 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c" event={"ID":"90fa8380-0306-4dee-91d7-77fc4a573610","Type":"ContainerDied","Data":"ff3ee17dce69c9a0266ebc3a61343f7e9ce38966b46111fd847ee1bf2ae51477"} Jul 10 19:04:16.890739 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:16.890698 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8t67g" event={"ID":"78ea5c71-2618-4053-8797-4ca3ad24778e","Type":"ContainerStarted","Data":"5e6dae65f430bf34c791ed6b225e90c8e196e67e5d05a900aee98ef77899016a"} Jul 10 19:04:16.891265 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:16.890778 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8t67g" Jul 10 19:04:16.909853 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:16.909794 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-8t67g" podStartSLOduration=1.9404590609999999 podStartE2EDuration="4.909773308s" totalImagesPullingTime="2.969314247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:04:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:04:16.90895811 +0000 UTC m=+223.510100246" watchObservedRunningTime="2026-07-10 19:04:16.909773308 +0000 UTC m=+223.510915444" Jul 10 19:04:17.359452 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:17.359411 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:17.457324 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:17.457286 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:17.896804 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:17.896765 2561 generic.go:356] "Generic (PLEG): container finished" podID="90fa8380-0306-4dee-91d7-77fc4a573610" containerID="e246ebc4d9802964479fb6ff5bd846d5b7bb86a20dc603e5a5c3209117cac76a" exitCode=0 Jul 10 19:04:17.897236 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:17.896855 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c" event={"ID":"90fa8380-0306-4dee-91d7-77fc4a573610","Type":"ContainerDied","Data":"e246ebc4d9802964479fb6ff5bd846d5b7bb86a20dc603e5a5c3209117cac76a"} Jul 10 19:04:18.902764 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:18.902730 2561 generic.go:356] "Generic (PLEG): container finished" podID="90fa8380-0306-4dee-91d7-77fc4a573610" containerID="51ef5a1b8046225dad5e83c75a4ee05d9ba86d80a4bc1e42956bc5d440f025b4" exitCode=0 Jul 10 19:04:18.903200 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:18.902821 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c" event={"ID":"90fa8380-0306-4dee-91d7-77fc4a573610","Type":"ContainerDied","Data":"51ef5a1b8046225dad5e83c75a4ee05d9ba86d80a4bc1e42956bc5d440f025b4"} Jul 10 19:04:20.024765 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:20.024743 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c" Jul 10 19:04:20.055490 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:20.055462 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bss6w\" (UniqueName: \"kubernetes.io/projected/90fa8380-0306-4dee-91d7-77fc4a573610-kube-api-access-bss6w\") pod \"90fa8380-0306-4dee-91d7-77fc4a573610\" (UID: \"90fa8380-0306-4dee-91d7-77fc4a573610\") " Jul 10 19:04:20.055632 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:20.055520 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90fa8380-0306-4dee-91d7-77fc4a573610-util\") pod \"90fa8380-0306-4dee-91d7-77fc4a573610\" (UID: \"90fa8380-0306-4dee-91d7-77fc4a573610\") " Jul 10 19:04:20.055632 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:20.055563 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90fa8380-0306-4dee-91d7-77fc4a573610-bundle\") pod \"90fa8380-0306-4dee-91d7-77fc4a573610\" (UID: \"90fa8380-0306-4dee-91d7-77fc4a573610\") " Jul 10 19:04:20.056032 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:20.056002 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90fa8380-0306-4dee-91d7-77fc4a573610-bundle" (OuterVolumeSpecName: "bundle") pod "90fa8380-0306-4dee-91d7-77fc4a573610" (UID: "90fa8380-0306-4dee-91d7-77fc4a573610"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:04:20.057787 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:20.057764 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90fa8380-0306-4dee-91d7-77fc4a573610-kube-api-access-bss6w" (OuterVolumeSpecName: "kube-api-access-bss6w") pod "90fa8380-0306-4dee-91d7-77fc4a573610" (UID: "90fa8380-0306-4dee-91d7-77fc4a573610"). InnerVolumeSpecName "kube-api-access-bss6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:04:20.062059 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:20.062024 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90fa8380-0306-4dee-91d7-77fc4a573610-util" (OuterVolumeSpecName: "util") pod "90fa8380-0306-4dee-91d7-77fc4a573610" (UID: "90fa8380-0306-4dee-91d7-77fc4a573610"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:04:20.156208 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:20.156170 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bss6w\" (UniqueName: \"kubernetes.io/projected/90fa8380-0306-4dee-91d7-77fc4a573610-kube-api-access-bss6w\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:04:20.156208 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:20.156203 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90fa8380-0306-4dee-91d7-77fc4a573610-util\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:04:20.156208 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:20.156219 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90fa8380-0306-4dee-91d7-77fc4a573610-bundle\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:04:20.912296 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:20.912267 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c" Jul 10 19:04:20.912296 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:20.912278 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f5w25c" event={"ID":"90fa8380-0306-4dee-91d7-77fc4a573610","Type":"ContainerDied","Data":"86427ed72d392b1dbadb397264250a8ba45042ad33a8617bf3f5a006150a105d"} Jul 10 19:04:20.912510 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:20.912358 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86427ed72d392b1dbadb397264250a8ba45042ad33a8617bf3f5a006150a105d" Jul 10 19:04:21.912768 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:21.912715 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:22.899556 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:22.899528 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8t67g" Jul 10 19:04:23.811183 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:23.811149 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:23.885056 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:23.885020 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:27.162173 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:27.162132 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:27.263525 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:27.263490 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:28.860712 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:28.860683 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzgx6"] Jul 10 19:04:28.861129 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:28.860997 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90fa8380-0306-4dee-91d7-77fc4a573610" containerName="util" Jul 10 19:04:28.861129 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:28.861010 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="90fa8380-0306-4dee-91d7-77fc4a573610" containerName="util" Jul 10 19:04:28.861129 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:28.861020 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90fa8380-0306-4dee-91d7-77fc4a573610" containerName="extract" Jul 10 19:04:28.861129 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:28.861025 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="90fa8380-0306-4dee-91d7-77fc4a573610" containerName="extract" Jul 10 19:04:28.861129 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:28.861038 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90fa8380-0306-4dee-91d7-77fc4a573610" containerName="pull" Jul 10 19:04:28.861129 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:28.861043 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="90fa8380-0306-4dee-91d7-77fc4a573610" containerName="pull" Jul 10 19:04:28.861129 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:28.861090 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="90fa8380-0306-4dee-91d7-77fc4a573610" containerName="extract" Jul 10 19:04:28.862929 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:28.862914 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzgx6" Jul 10 19:04:28.865573 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:28.865552 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 19:04:28.865710 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:28.865692 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 19:04:28.866610 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:28.866594 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-66wtx\"" Jul 10 19:04:28.873272 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:28.873247 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzgx6"] Jul 10 19:04:28.918087 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:28.918049 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8ljk\" (UniqueName: \"kubernetes.io/projected/b234f450-466c-46b3-9380-556e8109a117-kube-api-access-f8ljk\") pod \"openshift-lws-operator-fd8ccff4c-gzgx6\" (UID: \"b234f450-466c-46b3-9380-556e8109a117\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzgx6" Jul 10 19:04:28.918087 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:28.918089 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b234f450-466c-46b3-9380-556e8109a117-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gzgx6\" (UID: \"b234f450-466c-46b3-9380-556e8109a117\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzgx6" Jul 10 19:04:29.018686 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:29.018642 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8ljk\" (UniqueName: \"kubernetes.io/projected/b234f450-466c-46b3-9380-556e8109a117-kube-api-access-f8ljk\") pod \"openshift-lws-operator-fd8ccff4c-gzgx6\" (UID: \"b234f450-466c-46b3-9380-556e8109a117\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzgx6" Jul 10 19:04:29.018686 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:29.018690 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b234f450-466c-46b3-9380-556e8109a117-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gzgx6\" (UID: \"b234f450-466c-46b3-9380-556e8109a117\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzgx6" Jul 10 19:04:29.019109 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:29.019085 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b234f450-466c-46b3-9380-556e8109a117-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gzgx6\" (UID: \"b234f450-466c-46b3-9380-556e8109a117\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzgx6" Jul 10 19:04:29.026774 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:29.026753 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8ljk\" (UniqueName: \"kubernetes.io/projected/b234f450-466c-46b3-9380-556e8109a117-kube-api-access-f8ljk\") pod \"openshift-lws-operator-fd8ccff4c-gzgx6\" (UID: \"b234f450-466c-46b3-9380-556e8109a117\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzgx6" Jul 10 19:04:29.171534 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:29.171441 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzgx6" Jul 10 19:04:29.307262 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:29.307228 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzgx6"] Jul 10 19:04:29.310759 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:04:29.310735 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb234f450_466c_46b3_9380_556e8109a117.slice/crio-083194298bfd269aae258ee8f2958e7896a682be197fe10b0922fb8f0b29f141 WatchSource:0}: Error finding container 083194298bfd269aae258ee8f2958e7896a682be197fe10b0922fb8f0b29f141: Status 404 returned error can't find the container with id 083194298bfd269aae258ee8f2958e7896a682be197fe10b0922fb8f0b29f141 Jul 10 19:04:29.948153 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:29.948076 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzgx6" event={"ID":"b234f450-466c-46b3-9380-556e8109a117","Type":"ContainerStarted","Data":"083194298bfd269aae258ee8f2958e7896a682be197fe10b0922fb8f0b29f141"} Jul 10 19:04:31.962165 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:31.962112 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzgx6" event={"ID":"b234f450-466c-46b3-9380-556e8109a117","Type":"ContainerStarted","Data":"bde5234c8f9d1cc5afc39d2daee61fd25279c4f538315314f8ab272ddbcdf929"} Jul 10 19:04:31.975586 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:31.975537 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gzgx6" podStartSLOduration=2.326105598 podStartE2EDuration="3.975522467s" totalImagesPullingTime="1.649416869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:04:31.974876754 +0000 UTC m=+238.576018885" watchObservedRunningTime="2026-07-10 19:04:31.975522467 +0000 UTC m=+238.576664601" Jul 10 19:04:32.769452 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:32.769415 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:32.869104 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:32.869068 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:34.653697 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:34.651254 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq"] Jul 10 19:04:34.654896 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:34.654874 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq" Jul 10 19:04:34.657298 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:34.657274 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 10 19:04:34.657420 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:34.657394 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 10 19:04:34.658293 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:34.658279 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-hkrss\"" Jul 10 19:04:34.662156 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:34.662135 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq"] Jul 10 19:04:34.762605 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:34.762569 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53ab0c38-2814-462c-9540-6b6ebe54c111-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq\" (UID: \"53ab0c38-2814-462c-9540-6b6ebe54c111\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq" Jul 10 19:04:34.762775 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:34.762625 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chqx9\" (UniqueName: \"kubernetes.io/projected/53ab0c38-2814-462c-9540-6b6ebe54c111-kube-api-access-chqx9\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq\" (UID: \"53ab0c38-2814-462c-9540-6b6ebe54c111\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq" Jul 10 19:04:34.762775 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:34.762718 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53ab0c38-2814-462c-9540-6b6ebe54c111-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq\" (UID: \"53ab0c38-2814-462c-9540-6b6ebe54c111\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq" Jul 10 19:04:34.864032 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:34.863992 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53ab0c38-2814-462c-9540-6b6ebe54c111-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq\" (UID: \"53ab0c38-2814-462c-9540-6b6ebe54c111\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq" Jul 10 19:04:34.864229 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:34.864043 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53ab0c38-2814-462c-9540-6b6ebe54c111-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq\" (UID: \"53ab0c38-2814-462c-9540-6b6ebe54c111\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq" Jul 10 19:04:34.864229 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:34.864086 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chqx9\" (UniqueName: \"kubernetes.io/projected/53ab0c38-2814-462c-9540-6b6ebe54c111-kube-api-access-chqx9\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq\" (UID: \"53ab0c38-2814-462c-9540-6b6ebe54c111\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq" Jul 10 19:04:34.864394 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:34.864373 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53ab0c38-2814-462c-9540-6b6ebe54c111-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq\" (UID: \"53ab0c38-2814-462c-9540-6b6ebe54c111\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq" Jul 10 19:04:34.864432 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:34.864396 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53ab0c38-2814-462c-9540-6b6ebe54c111-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq\" (UID: \"53ab0c38-2814-462c-9540-6b6ebe54c111\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq" Jul 10 19:04:34.873228 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:34.873203 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chqx9\" (UniqueName: \"kubernetes.io/projected/53ab0c38-2814-462c-9540-6b6ebe54c111-kube-api-access-chqx9\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq\" (UID: \"53ab0c38-2814-462c-9540-6b6ebe54c111\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq" Jul 10 19:04:34.965034 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:34.964956 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq" Jul 10 19:04:34.996233 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:34.996204 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:35.096711 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:35.096687 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq"] Jul 10 19:04:35.098546 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:04:35.098517 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ab0c38_2814_462c_9540_6b6ebe54c111.slice/crio-aaef95c76eb59f6b711b21b5afef2b2ecb975cde362116176d0a3bbc42f394af WatchSource:0}: Error finding container aaef95c76eb59f6b711b21b5afef2b2ecb975cde362116176d0a3bbc42f394af: Status 404 returned error can't find the container with id aaef95c76eb59f6b711b21b5afef2b2ecb975cde362116176d0a3bbc42f394af Jul 10 19:04:35.975225 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:35.975185 2561 generic.go:356] "Generic (PLEG): container finished" podID="53ab0c38-2814-462c-9540-6b6ebe54c111" containerID="35808e8faaa62ad392f3a7160f289fa6318df9c4879c99187f719d634ae35b31" exitCode=0 Jul 10 19:04:35.975662 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:35.975272 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq" event={"ID":"53ab0c38-2814-462c-9540-6b6ebe54c111","Type":"ContainerDied","Data":"35808e8faaa62ad392f3a7160f289fa6318df9c4879c99187f719d634ae35b31"} Jul 10 19:04:35.975662 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:35.975330 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq" event={"ID":"53ab0c38-2814-462c-9540-6b6ebe54c111","Type":"ContainerStarted","Data":"aaef95c76eb59f6b711b21b5afef2b2ecb975cde362116176d0a3bbc42f394af"} Jul 10 19:04:36.981830 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:36.981747 2561 generic.go:356] "Generic (PLEG): container finished" podID="53ab0c38-2814-462c-9540-6b6ebe54c111" containerID="a04f1339dc7c8f4566db8aa03a7c6bf5c9f013b6edd480576a59e30e7aecd1e7" exitCode=0 Jul 10 19:04:36.981830 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:36.981809 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq" event={"ID":"53ab0c38-2814-462c-9540-6b6ebe54c111","Type":"ContainerDied","Data":"a04f1339dc7c8f4566db8aa03a7c6bf5c9f013b6edd480576a59e30e7aecd1e7"} Jul 10 19:04:37.987827 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:37.987792 2561 generic.go:356] "Generic (PLEG): container finished" podID="53ab0c38-2814-462c-9540-6b6ebe54c111" containerID="5c8c5bac62f8c9f566d759f84865353c39136e3b233141ce3cb8d34c7427bfc4" exitCode=0 Jul 10 19:04:37.988233 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:37.987859 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq" event={"ID":"53ab0c38-2814-462c-9540-6b6ebe54c111","Type":"ContainerDied","Data":"5c8c5bac62f8c9f566d759f84865353c39136e3b233141ce3cb8d34c7427bfc4"} Jul 10 19:04:39.116999 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:39.116976 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq" Jul 10 19:04:39.197116 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:39.197077 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chqx9\" (UniqueName: \"kubernetes.io/projected/53ab0c38-2814-462c-9540-6b6ebe54c111-kube-api-access-chqx9\") pod \"53ab0c38-2814-462c-9540-6b6ebe54c111\" (UID: \"53ab0c38-2814-462c-9540-6b6ebe54c111\") " Jul 10 19:04:39.197116 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:39.197118 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53ab0c38-2814-462c-9540-6b6ebe54c111-util\") pod \"53ab0c38-2814-462c-9540-6b6ebe54c111\" (UID: \"53ab0c38-2814-462c-9540-6b6ebe54c111\") " Jul 10 19:04:39.197342 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:39.197160 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53ab0c38-2814-462c-9540-6b6ebe54c111-bundle\") pod \"53ab0c38-2814-462c-9540-6b6ebe54c111\" (UID: \"53ab0c38-2814-462c-9540-6b6ebe54c111\") " Jul 10 19:04:39.197864 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:39.197838 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53ab0c38-2814-462c-9540-6b6ebe54c111-bundle" (OuterVolumeSpecName: "bundle") pod "53ab0c38-2814-462c-9540-6b6ebe54c111" (UID: "53ab0c38-2814-462c-9540-6b6ebe54c111"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:04:39.199311 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:39.199276 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ab0c38-2814-462c-9540-6b6ebe54c111-kube-api-access-chqx9" (OuterVolumeSpecName: "kube-api-access-chqx9") pod "53ab0c38-2814-462c-9540-6b6ebe54c111" (UID: "53ab0c38-2814-462c-9540-6b6ebe54c111"). InnerVolumeSpecName "kube-api-access-chqx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:04:39.202853 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:39.202833 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53ab0c38-2814-462c-9540-6b6ebe54c111-util" (OuterVolumeSpecName: "util") pod "53ab0c38-2814-462c-9540-6b6ebe54c111" (UID: "53ab0c38-2814-462c-9540-6b6ebe54c111"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:04:39.298309 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:39.298279 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chqx9\" (UniqueName: \"kubernetes.io/projected/53ab0c38-2814-462c-9540-6b6ebe54c111-kube-api-access-chqx9\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:04:39.298309 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:39.298304 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53ab0c38-2814-462c-9540-6b6ebe54c111-util\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:04:39.298309 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:39.298316 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53ab0c38-2814-462c-9540-6b6ebe54c111-bundle\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:04:39.997350 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:39.997312 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq" event={"ID":"53ab0c38-2814-462c-9540-6b6ebe54c111","Type":"ContainerDied","Data":"aaef95c76eb59f6b711b21b5afef2b2ecb975cde362116176d0a3bbc42f394af"} Jul 10 19:04:39.997494 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:39.997393 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaef95c76eb59f6b711b21b5afef2b2ecb975cde362116176d0a3bbc42f394af" Jul 10 19:04:39.997494 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:39.997323 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afvzjbq" Jul 10 19:04:50.566260 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:50.566224 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:50.658004 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:50.657969 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:51.199430 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.199394 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:51.442824 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.442794 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:51.545606 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.545572 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:51.594165 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.594124 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc"] Jul 10 19:04:51.594698 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.594519 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53ab0c38-2814-462c-9540-6b6ebe54c111" containerName="extract" Jul 10 19:04:51.594698 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.594538 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="53ab0c38-2814-462c-9540-6b6ebe54c111" containerName="extract" Jul 10 19:04:51.594698 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.594550 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53ab0c38-2814-462c-9540-6b6ebe54c111" containerName="util" Jul 10 19:04:51.594698 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.594558 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="53ab0c38-2814-462c-9540-6b6ebe54c111" containerName="util" Jul 10 19:04:51.594698 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.594581 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53ab0c38-2814-462c-9540-6b6ebe54c111" containerName="pull" Jul 10 19:04:51.594698 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.594588 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="53ab0c38-2814-462c-9540-6b6ebe54c111" containerName="pull" Jul 10 19:04:51.594698 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.594657 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="53ab0c38-2814-462c-9540-6b6ebe54c111" containerName="extract" Jul 10 19:04:51.598886 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.598863 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" Jul 10 19:04:51.602227 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.602206 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 19:04:51.602395 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.602375 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 19:04:51.603231 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.603199 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 19:04:51.603430 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.603413 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 19:04:51.603767 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.603752 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lhpvl\"" Jul 10 19:04:51.614396 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.614375 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc"] Jul 10 19:04:51.692440 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.692400 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22acb39a-d928-4715-a844-945c72839563-webhook-cert\") pod \"opendatahub-operator-controller-manager-7668cd484b-9qvkc\" (UID: \"22acb39a-d928-4715-a844-945c72839563\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" Jul 10 19:04:51.692440 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.692444 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22acb39a-d928-4715-a844-945c72839563-manifests\") pod \"opendatahub-operator-controller-manager-7668cd484b-9qvkc\" (UID: \"22acb39a-d928-4715-a844-945c72839563\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" Jul 10 19:04:51.692638 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.692506 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22acb39a-d928-4715-a844-945c72839563-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7668cd484b-9qvkc\" (UID: \"22acb39a-d928-4715-a844-945c72839563\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" Jul 10 19:04:51.692638 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.692563 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22acb39a-d928-4715-a844-945c72839563-tmp\") pod \"opendatahub-operator-controller-manager-7668cd484b-9qvkc\" (UID: \"22acb39a-d928-4715-a844-945c72839563\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" Jul 10 19:04:51.692638 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.692582 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tjmf\" (UniqueName: \"kubernetes.io/projected/22acb39a-d928-4715-a844-945c72839563-kube-api-access-5tjmf\") pod \"opendatahub-operator-controller-manager-7668cd484b-9qvkc\" (UID: \"22acb39a-d928-4715-a844-945c72839563\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" Jul 10 19:04:51.738532 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.738498 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:51.793850 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.793821 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22acb39a-d928-4715-a844-945c72839563-tmp\") pod \"opendatahub-operator-controller-manager-7668cd484b-9qvkc\" (UID: \"22acb39a-d928-4715-a844-945c72839563\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" Jul 10 19:04:51.793993 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.793855 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tjmf\" (UniqueName: \"kubernetes.io/projected/22acb39a-d928-4715-a844-945c72839563-kube-api-access-5tjmf\") pod \"opendatahub-operator-controller-manager-7668cd484b-9qvkc\" (UID: \"22acb39a-d928-4715-a844-945c72839563\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" Jul 10 19:04:51.793993 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.793896 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22acb39a-d928-4715-a844-945c72839563-webhook-cert\") pod \"opendatahub-operator-controller-manager-7668cd484b-9qvkc\" (UID: \"22acb39a-d928-4715-a844-945c72839563\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" Jul 10 19:04:51.793993 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.793931 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22acb39a-d928-4715-a844-945c72839563-manifests\") pod \"opendatahub-operator-controller-manager-7668cd484b-9qvkc\" (UID: \"22acb39a-d928-4715-a844-945c72839563\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" Jul 10 19:04:51.794148 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.794002 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22acb39a-d928-4715-a844-945c72839563-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7668cd484b-9qvkc\" (UID: \"22acb39a-d928-4715-a844-945c72839563\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" Jul 10 19:04:51.794357 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.794335 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22acb39a-d928-4715-a844-945c72839563-manifests\") pod \"opendatahub-operator-controller-manager-7668cd484b-9qvkc\" (UID: \"22acb39a-d928-4715-a844-945c72839563\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" Jul 10 19:04:51.796495 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.796432 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22acb39a-d928-4715-a844-945c72839563-tmp\") pod \"opendatahub-operator-controller-manager-7668cd484b-9qvkc\" (UID: \"22acb39a-d928-4715-a844-945c72839563\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" Jul 10 19:04:51.796495 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.796448 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22acb39a-d928-4715-a844-945c72839563-webhook-cert\") pod \"opendatahub-operator-controller-manager-7668cd484b-9qvkc\" (UID: \"22acb39a-d928-4715-a844-945c72839563\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" Jul 10 19:04:51.796651 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.796557 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22acb39a-d928-4715-a844-945c72839563-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7668cd484b-9qvkc\" (UID: \"22acb39a-d928-4715-a844-945c72839563\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" Jul 10 19:04:51.802777 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.802753 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tjmf\" (UniqueName: \"kubernetes.io/projected/22acb39a-d928-4715-a844-945c72839563-kube-api-access-5tjmf\") pod \"opendatahub-operator-controller-manager-7668cd484b-9qvkc\" (UID: \"22acb39a-d928-4715-a844-945c72839563\") " pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" Jul 10 19:04:51.831175 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.831141 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:51.909039 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.909006 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" Jul 10 19:04:51.931267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:51.931236 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:52.035502 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:52.035466 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:52.052383 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:52.052312 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc"] Jul 10 19:04:52.055896 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:04:52.055868 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22acb39a_d928_4715_a844_945c72839563.slice/crio-7627571b0767be30168c0eed8d8a41fe39fdbb8e871a51624b16e5b94cdf5ea7 WatchSource:0}: Error finding container 7627571b0767be30168c0eed8d8a41fe39fdbb8e871a51624b16e5b94cdf5ea7: Status 404 returned error can't find the container with id 7627571b0767be30168c0eed8d8a41fe39fdbb8e871a51624b16e5b94cdf5ea7 Jul 10 19:04:52.140259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:52.140222 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:53.053329 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:53.053261 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" event={"ID":"22acb39a-d928-4715-a844-945c72839563","Type":"ContainerStarted","Data":"7627571b0767be30168c0eed8d8a41fe39fdbb8e871a51624b16e5b94cdf5ea7"} Jul 10 19:04:56.068460 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:56.068422 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" event={"ID":"22acb39a-d928-4715-a844-945c72839563","Type":"ContainerStarted","Data":"0649539152b87a3d5f8069699aa11865ab5c2bfc8d5ebb5dfb5e8e17a3e66bff"} Jul 10 19:04:57.074117 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:57.074080 2561 generic.go:356] "Generic (PLEG): container finished" podID="22acb39a-d928-4715-a844-945c72839563" containerID="0649539152b87a3d5f8069699aa11865ab5c2bfc8d5ebb5dfb5e8e17a3e66bff" exitCode=0 Jul 10 19:04:57.074493 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:57.074174 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" event={"ID":"22acb39a-d928-4715-a844-945c72839563","Type":"ContainerDied","Data":"0649539152b87a3d5f8069699aa11865ab5c2bfc8d5ebb5dfb5e8e17a3e66bff"} Jul 10 19:04:58.081405 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:58.081366 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" event={"ID":"22acb39a-d928-4715-a844-945c72839563","Type":"ContainerStarted","Data":"7d7ed086d8acf61aee3e42f0a1c9de78aa17150141f934bbf2cc43329380f707"} Jul 10 19:04:58.081787 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:58.081469 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" Jul 10 19:04:58.113131 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:58.113082 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" podStartSLOduration=2.528704602 podStartE2EDuration="7.11306637s" totalImagesPullingTime="3.577671979s" totalInitContainerRuntime="1.006689789s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:04:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:04:58.111228152 +0000 UTC m=+264.712370298" watchObservedRunningTime="2026-07-10 19:04:58.11306637 +0000 UTC m=+264.714208504" Jul 10 19:04:59.501176 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:59.501142 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:04:59.604635 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:04:59.604601 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:03.622478 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:03.622441 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:03.726596 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:03.726557 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:09.087155 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:09.087125 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7668cd484b-9qvkc" Jul 10 19:05:10.821396 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:10.821358 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:10.964532 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:10.964499 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:11.558188 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:11.558151 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:11.655716 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:11.655679 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:11.968386 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:11.968299 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:12.237590 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:12.237490 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:12.412254 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:12.412218 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:16.769367 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:16.769329 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:16.839412 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:16.839377 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58b4fc6b9d-7s88n"] Jul 10 19:05:16.844640 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:16.844618 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-7s88n" Jul 10 19:05:16.848341 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:16.848314 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 19:05:16.848471 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:16.848400 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 19:05:16.848539 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:16.848519 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 19:05:16.848599 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:16.848522 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 19:05:16.849381 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:16.849364 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wb8gg\"" Jul 10 19:05:16.856536 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:16.856517 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58b4fc6b9d-7s88n"] Jul 10 19:05:16.971543 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:16.971503 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:16.990618 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:16.990583 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36c31443-af47-4c59-a1d4-a00aa0f29dee-tls-certs\") pod \"kube-auth-proxy-58b4fc6b9d-7s88n\" (UID: \"36c31443-af47-4c59-a1d4-a00aa0f29dee\") " pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-7s88n" Jul 10 19:05:16.990742 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:16.990622 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps9np\" (UniqueName: \"kubernetes.io/projected/36c31443-af47-4c59-a1d4-a00aa0f29dee-kube-api-access-ps9np\") pod \"kube-auth-proxy-58b4fc6b9d-7s88n\" (UID: \"36c31443-af47-4c59-a1d4-a00aa0f29dee\") " pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-7s88n" Jul 10 19:05:16.990742 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:16.990660 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36c31443-af47-4c59-a1d4-a00aa0f29dee-tmp\") pod \"kube-auth-proxy-58b4fc6b9d-7s88n\" (UID: \"36c31443-af47-4c59-a1d4-a00aa0f29dee\") " pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-7s88n" Jul 10 19:05:17.091361 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:17.091277 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36c31443-af47-4c59-a1d4-a00aa0f29dee-tls-certs\") pod \"kube-auth-proxy-58b4fc6b9d-7s88n\" (UID: \"36c31443-af47-4c59-a1d4-a00aa0f29dee\") " pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-7s88n" Jul 10 19:05:17.091361 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:17.091311 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps9np\" (UniqueName: \"kubernetes.io/projected/36c31443-af47-4c59-a1d4-a00aa0f29dee-kube-api-access-ps9np\") pod \"kube-auth-proxy-58b4fc6b9d-7s88n\" (UID: \"36c31443-af47-4c59-a1d4-a00aa0f29dee\") " pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-7s88n" Jul 10 19:05:17.091361 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:17.091348 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36c31443-af47-4c59-a1d4-a00aa0f29dee-tmp\") pod \"kube-auth-proxy-58b4fc6b9d-7s88n\" (UID: \"36c31443-af47-4c59-a1d4-a00aa0f29dee\") " pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-7s88n" Jul 10 19:05:17.094244 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:17.094209 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/36c31443-af47-4c59-a1d4-a00aa0f29dee-tmp\") pod \"kube-auth-proxy-58b4fc6b9d-7s88n\" (UID: \"36c31443-af47-4c59-a1d4-a00aa0f29dee\") " pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-7s88n" Jul 10 19:05:17.094449 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:17.094429 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36c31443-af47-4c59-a1d4-a00aa0f29dee-tls-certs\") pod \"kube-auth-proxy-58b4fc6b9d-7s88n\" (UID: \"36c31443-af47-4c59-a1d4-a00aa0f29dee\") " pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-7s88n" Jul 10 19:05:17.106265 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:17.106229 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps9np\" (UniqueName: \"kubernetes.io/projected/36c31443-af47-4c59-a1d4-a00aa0f29dee-kube-api-access-ps9np\") pod \"kube-auth-proxy-58b4fc6b9d-7s88n\" (UID: \"36c31443-af47-4c59-a1d4-a00aa0f29dee\") " pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-7s88n" Jul 10 19:05:17.154492 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:17.154458 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-7s88n" Jul 10 19:05:17.328508 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:17.328483 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58b4fc6b9d-7s88n"] Jul 10 19:05:17.329899 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:05:17.329872 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c31443_af47_4c59_a1d4_a00aa0f29dee.slice/crio-b2931abd5c6d5b7fc29d5c90e38a34b59e8657596dd7bb62c1ec036751db881d WatchSource:0}: Error finding container b2931abd5c6d5b7fc29d5c90e38a34b59e8657596dd7bb62c1ec036751db881d: Status 404 returned error can't find the container with id b2931abd5c6d5b7fc29d5c90e38a34b59e8657596dd7bb62c1ec036751db881d Jul 10 19:05:18.167574 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:18.167538 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-7s88n" event={"ID":"36c31443-af47-4c59-a1d4-a00aa0f29dee","Type":"ContainerStarted","Data":"b2931abd5c6d5b7fc29d5c90e38a34b59e8657596dd7bb62c1ec036751db881d"} Jul 10 19:05:18.708703 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:18.708669 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:18.829167 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:18.829132 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:19.595370 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.595336 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk"] Jul 10 19:05:19.597532 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.597515 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk" Jul 10 19:05:19.601093 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.601070 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 19:05:19.601093 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.601070 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2tct4\"" Jul 10 19:05:19.601371 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.601331 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 19:05:19.601371 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.601329 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 19:05:19.617799 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.617773 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk"] Jul 10 19:05:19.713471 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.713434 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c87cd792-cd69-47b3-bebe-213827cbb39d-metrics-cert\") pod \"lws-controller-manager-6d9b94cf9d-lbwrk\" (UID: \"c87cd792-cd69-47b3-bebe-213827cbb39d\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk" Jul 10 19:05:19.713471 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.713473 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c87cd792-cd69-47b3-bebe-213827cbb39d-manager-config\") pod \"lws-controller-manager-6d9b94cf9d-lbwrk\" (UID: \"c87cd792-cd69-47b3-bebe-213827cbb39d\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk" Jul 10 19:05:19.713683 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.713500 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c87cd792-cd69-47b3-bebe-213827cbb39d-cert\") pod \"lws-controller-manager-6d9b94cf9d-lbwrk\" (UID: \"c87cd792-cd69-47b3-bebe-213827cbb39d\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk" Jul 10 19:05:19.713683 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.713572 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42rm5\" (UniqueName: \"kubernetes.io/projected/c87cd792-cd69-47b3-bebe-213827cbb39d-kube-api-access-42rm5\") pod \"lws-controller-manager-6d9b94cf9d-lbwrk\" (UID: \"c87cd792-cd69-47b3-bebe-213827cbb39d\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk" Jul 10 19:05:19.814501 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.814462 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c87cd792-cd69-47b3-bebe-213827cbb39d-metrics-cert\") pod \"lws-controller-manager-6d9b94cf9d-lbwrk\" (UID: \"c87cd792-cd69-47b3-bebe-213827cbb39d\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk" Jul 10 19:05:19.814678 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.814509 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c87cd792-cd69-47b3-bebe-213827cbb39d-manager-config\") pod \"lws-controller-manager-6d9b94cf9d-lbwrk\" (UID: \"c87cd792-cd69-47b3-bebe-213827cbb39d\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk" Jul 10 19:05:19.814678 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.814549 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c87cd792-cd69-47b3-bebe-213827cbb39d-cert\") pod \"lws-controller-manager-6d9b94cf9d-lbwrk\" (UID: \"c87cd792-cd69-47b3-bebe-213827cbb39d\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk" Jul 10 19:05:19.814678 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.814578 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42rm5\" (UniqueName: \"kubernetes.io/projected/c87cd792-cd69-47b3-bebe-213827cbb39d-kube-api-access-42rm5\") pod \"lws-controller-manager-6d9b94cf9d-lbwrk\" (UID: \"c87cd792-cd69-47b3-bebe-213827cbb39d\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk" Jul 10 19:05:19.815415 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.815390 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c87cd792-cd69-47b3-bebe-213827cbb39d-manager-config\") pod \"lws-controller-manager-6d9b94cf9d-lbwrk\" (UID: \"c87cd792-cd69-47b3-bebe-213827cbb39d\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk" Jul 10 19:05:19.817509 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.817481 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c87cd792-cd69-47b3-bebe-213827cbb39d-cert\") pod \"lws-controller-manager-6d9b94cf9d-lbwrk\" (UID: \"c87cd792-cd69-47b3-bebe-213827cbb39d\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk" Jul 10 19:05:19.817646 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.817601 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c87cd792-cd69-47b3-bebe-213827cbb39d-metrics-cert\") pod \"lws-controller-manager-6d9b94cf9d-lbwrk\" (UID: \"c87cd792-cd69-47b3-bebe-213827cbb39d\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk" Jul 10 19:05:19.829502 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.829478 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42rm5\" (UniqueName: \"kubernetes.io/projected/c87cd792-cd69-47b3-bebe-213827cbb39d-kube-api-access-42rm5\") pod \"lws-controller-manager-6d9b94cf9d-lbwrk\" (UID: \"c87cd792-cd69-47b3-bebe-213827cbb39d\") " pod="openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk" Jul 10 19:05:19.907433 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:19.907342 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk" Jul 10 19:05:20.510336 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:20.510309 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk"] Jul 10 19:05:20.512107 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:05:20.512078 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87cd792_cd69_47b3_bebe_213827cbb39d.slice/crio-59571017274adba4e3bf877600862ded8e3d3cddc98ae71eb3d90fb7bd69371d WatchSource:0}: Error finding container 59571017274adba4e3bf877600862ded8e3d3cddc98ae71eb3d90fb7bd69371d: Status 404 returned error can't find the container with id 59571017274adba4e3bf877600862ded8e3d3cddc98ae71eb3d90fb7bd69371d Jul 10 19:05:20.694759 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:20.694640 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:21.182891 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:21.182856 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-7s88n" event={"ID":"36c31443-af47-4c59-a1d4-a00aa0f29dee","Type":"ContainerStarted","Data":"4a7c8b713c0205d9adb4c949e8b5c170cca541acaf68a036d9bc974a6026a139"} Jul 10 19:05:21.184095 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:21.184071 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk" event={"ID":"c87cd792-cd69-47b3-bebe-213827cbb39d","Type":"ContainerStarted","Data":"59571017274adba4e3bf877600862ded8e3d3cddc98ae71eb3d90fb7bd69371d"} Jul 10 19:05:21.210707 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:21.210652 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58b4fc6b9d-7s88n" podStartSLOduration=2.139484943 podStartE2EDuration="5.210637399s" totalImagesPullingTime="3.071152456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:21.208602381 +0000 UTC m=+287.809744526" watchObservedRunningTime="2026-07-10 19:05:21.210637399 +0000 UTC m=+287.811779533" Jul 10 19:05:23.196239 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:23.196191 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk" event={"ID":"c87cd792-cd69-47b3-bebe-213827cbb39d","Type":"ContainerStarted","Data":"0433633b4928cb03b5e6dc1fcde0c360e4eb2246cc1ae2d5f5e0ae260231ed2a"} Jul 10 19:05:23.196670 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:23.196334 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk" Jul 10 19:05:23.220947 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:23.220890 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk" podStartSLOduration=2.516927254 podStartE2EDuration="4.220873964s" totalImagesPullingTime="1.70394671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:05:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:05:23.220599476 +0000 UTC m=+289.821741611" watchObservedRunningTime="2026-07-10 19:05:23.220873964 +0000 UTC m=+289.822016500" Jul 10 19:05:23.241315 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:23.241283 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:23.786529 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:23.786497 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:23.914869 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:23.914835 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:24.017324 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:24.017281 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:33.841579 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:33.841546 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:05:33.846262 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:33.846238 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:05:33.849443 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:33.849411 2561 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 19:05:34.203356 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:34.203268 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6d9b94cf9d-lbwrk" Jul 10 19:05:39.076782 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:39.076743 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:39.314815 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:39.314767 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:39.414275 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:39.414183 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:39.636727 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:39.636689 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:05:40.114613 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:05:40.114578 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:08.031884 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:08.031855 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:08.110585 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:08.110549 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:12.437551 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:12.437517 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:12.576453 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:12.576421 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:12.693890 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:12.693804 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:17.972270 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:17.972239 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:18.047251 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:18.047221 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:27.876237 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:27.876199 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x"] Jul 10 19:06:27.879708 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:27.879692 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x" Jul 10 19:06:27.882682 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:27.882660 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 19:06:27.882682 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:27.882672 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 19:06:27.883704 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:27.883689 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-k59z6\"" Jul 10 19:06:27.892076 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:27.892053 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x"] Jul 10 19:06:28.037725 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.037675 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fb12101-ccd5-474e-b4b7-b05700be5463-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x\" (UID: \"1fb12101-ccd5-474e-b4b7-b05700be5463\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x" Jul 10 19:06:28.037725 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.037732 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fb12101-ccd5-474e-b4b7-b05700be5463-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x\" (UID: \"1fb12101-ccd5-474e-b4b7-b05700be5463\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x" Jul 10 19:06:28.037971 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.037790 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmkmg\" (UniqueName: \"kubernetes.io/projected/1fb12101-ccd5-474e-b4b7-b05700be5463-kube-api-access-tmkmg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x\" (UID: \"1fb12101-ccd5-474e-b4b7-b05700be5463\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x" Jul 10 19:06:28.138675 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.138579 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fb12101-ccd5-474e-b4b7-b05700be5463-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x\" (UID: \"1fb12101-ccd5-474e-b4b7-b05700be5463\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x" Jul 10 19:06:28.138675 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.138637 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fb12101-ccd5-474e-b4b7-b05700be5463-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x\" (UID: \"1fb12101-ccd5-474e-b4b7-b05700be5463\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x" Jul 10 19:06:28.138675 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.138668 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmkmg\" (UniqueName: \"kubernetes.io/projected/1fb12101-ccd5-474e-b4b7-b05700be5463-kube-api-access-tmkmg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x\" (UID: \"1fb12101-ccd5-474e-b4b7-b05700be5463\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x" Jul 10 19:06:28.139008 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.138988 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fb12101-ccd5-474e-b4b7-b05700be5463-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x\" (UID: \"1fb12101-ccd5-474e-b4b7-b05700be5463\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x" Jul 10 19:06:28.139084 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.139062 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fb12101-ccd5-474e-b4b7-b05700be5463-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x\" (UID: \"1fb12101-ccd5-474e-b4b7-b05700be5463\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x" Jul 10 19:06:28.147853 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.147826 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmkmg\" (UniqueName: \"kubernetes.io/projected/1fb12101-ccd5-474e-b4b7-b05700be5463-kube-api-access-tmkmg\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x\" (UID: \"1fb12101-ccd5-474e-b4b7-b05700be5463\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x" Jul 10 19:06:28.188818 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.188783 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x" Jul 10 19:06:28.325518 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.325416 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x"] Jul 10 19:06:28.328211 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:06:28.328186 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb12101_ccd5_474e_b4b7_b05700be5463.slice/crio-cba2034726bbe391cdbf349be12e15f8cb608350cf52d68be5756735793a5ec2 WatchSource:0}: Error finding container cba2034726bbe391cdbf349be12e15f8cb608350cf52d68be5756735793a5ec2: Status 404 returned error can't find the container with id cba2034726bbe391cdbf349be12e15f8cb608350cf52d68be5756735793a5ec2 Jul 10 19:06:28.464899 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.464863 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9"] Jul 10 19:06:28.468154 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.468137 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9" Jul 10 19:06:28.475374 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.475348 2561 generic.go:356] "Generic (PLEG): container finished" podID="1fb12101-ccd5-474e-b4b7-b05700be5463" containerID="9656b9ee0f62470bea815212d15d9cb82595c6a748342cffd00cda18b544d7c6" exitCode=0 Jul 10 19:06:28.475525 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.475428 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x" event={"ID":"1fb12101-ccd5-474e-b4b7-b05700be5463","Type":"ContainerDied","Data":"9656b9ee0f62470bea815212d15d9cb82595c6a748342cffd00cda18b544d7c6"} Jul 10 19:06:28.475525 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.475473 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x" event={"ID":"1fb12101-ccd5-474e-b4b7-b05700be5463","Type":"ContainerStarted","Data":"cba2034726bbe391cdbf349be12e15f8cb608350cf52d68be5756735793a5ec2"} Jul 10 19:06:28.475885 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.475872 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:06:28.481348 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.481325 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9"] Jul 10 19:06:28.542361 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.542322 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a5081e81-09af-4b03-a57e-1f1e89455699-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9\" (UID: \"a5081e81-09af-4b03-a57e-1f1e89455699\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9" Jul 10 19:06:28.542540 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.542367 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a5081e81-09af-4b03-a57e-1f1e89455699-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9\" (UID: \"a5081e81-09af-4b03-a57e-1f1e89455699\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9" Jul 10 19:06:28.542540 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.542404 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrqfd\" (UniqueName: \"kubernetes.io/projected/a5081e81-09af-4b03-a57e-1f1e89455699-kube-api-access-mrqfd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9\" (UID: \"a5081e81-09af-4b03-a57e-1f1e89455699\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9" Jul 10 19:06:28.643257 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.643216 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a5081e81-09af-4b03-a57e-1f1e89455699-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9\" (UID: \"a5081e81-09af-4b03-a57e-1f1e89455699\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9" Jul 10 19:06:28.643257 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.643261 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a5081e81-09af-4b03-a57e-1f1e89455699-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9\" (UID: \"a5081e81-09af-4b03-a57e-1f1e89455699\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9" Jul 10 19:06:28.643506 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.643289 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrqfd\" (UniqueName: \"kubernetes.io/projected/a5081e81-09af-4b03-a57e-1f1e89455699-kube-api-access-mrqfd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9\" (UID: \"a5081e81-09af-4b03-a57e-1f1e89455699\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9" Jul 10 19:06:28.643693 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.643670 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a5081e81-09af-4b03-a57e-1f1e89455699-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9\" (UID: \"a5081e81-09af-4b03-a57e-1f1e89455699\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9" Jul 10 19:06:28.643737 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.643682 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a5081e81-09af-4b03-a57e-1f1e89455699-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9\" (UID: \"a5081e81-09af-4b03-a57e-1f1e89455699\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9" Jul 10 19:06:28.653196 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.653166 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrqfd\" (UniqueName: \"kubernetes.io/projected/a5081e81-09af-4b03-a57e-1f1e89455699-kube-api-access-mrqfd\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9\" (UID: \"a5081e81-09af-4b03-a57e-1f1e89455699\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9" Jul 10 19:06:28.812922 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.812886 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9" Jul 10 19:06:28.946752 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:28.946725 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9"] Jul 10 19:06:28.948757 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:06:28.948733 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5081e81_09af_4b03_a57e_1f1e89455699.slice/crio-bf36cd867d68800395149bddc861382cf7fd54d3c4420c4d7210cf292a2e78cd WatchSource:0}: Error finding container bf36cd867d68800395149bddc861382cf7fd54d3c4420c4d7210cf292a2e78cd: Status 404 returned error can't find the container with id bf36cd867d68800395149bddc861382cf7fd54d3c4420c4d7210cf292a2e78cd Jul 10 19:06:29.067268 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.067238 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd"] Jul 10 19:06:29.070482 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.070463 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd" Jul 10 19:06:29.083163 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.083131 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd"] Jul 10 19:06:29.148592 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.148510 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8631bef5-d160-4ae4-88b0-b137c25ced1e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd\" (UID: \"8631bef5-d160-4ae4-88b0-b137c25ced1e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd" Jul 10 19:06:29.148592 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.148574 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8631bef5-d160-4ae4-88b0-b137c25ced1e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd\" (UID: \"8631bef5-d160-4ae4-88b0-b137c25ced1e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd" Jul 10 19:06:29.148774 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.148597 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rcvn\" (UniqueName: \"kubernetes.io/projected/8631bef5-d160-4ae4-88b0-b137c25ced1e-kube-api-access-4rcvn\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd\" (UID: \"8631bef5-d160-4ae4-88b0-b137c25ced1e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd" Jul 10 19:06:29.250044 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.250000 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8631bef5-d160-4ae4-88b0-b137c25ced1e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd\" (UID: \"8631bef5-d160-4ae4-88b0-b137c25ced1e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd" Jul 10 19:06:29.250253 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.250079 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8631bef5-d160-4ae4-88b0-b137c25ced1e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd\" (UID: \"8631bef5-d160-4ae4-88b0-b137c25ced1e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd" Jul 10 19:06:29.250253 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.250111 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rcvn\" (UniqueName: \"kubernetes.io/projected/8631bef5-d160-4ae4-88b0-b137c25ced1e-kube-api-access-4rcvn\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd\" (UID: \"8631bef5-d160-4ae4-88b0-b137c25ced1e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd" Jul 10 19:06:29.250405 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.250383 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8631bef5-d160-4ae4-88b0-b137c25ced1e-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd\" (UID: \"8631bef5-d160-4ae4-88b0-b137c25ced1e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd" Jul 10 19:06:29.250405 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.250396 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8631bef5-d160-4ae4-88b0-b137c25ced1e-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd\" (UID: \"8631bef5-d160-4ae4-88b0-b137c25ced1e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd" Jul 10 19:06:29.259354 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.259329 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rcvn\" (UniqueName: \"kubernetes.io/projected/8631bef5-d160-4ae4-88b0-b137c25ced1e-kube-api-access-4rcvn\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd\" (UID: \"8631bef5-d160-4ae4-88b0-b137c25ced1e\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd" Jul 10 19:06:29.424302 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.424220 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd" Jul 10 19:06:29.484178 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.484144 2561 generic.go:356] "Generic (PLEG): container finished" podID="1fb12101-ccd5-474e-b4b7-b05700be5463" containerID="f13ad5b70a640cf89023d4647411ff3ddc76619ffa0b7fdc1a4eb12c0299084a" exitCode=0 Jul 10 19:06:29.484345 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.484178 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x" event={"ID":"1fb12101-ccd5-474e-b4b7-b05700be5463","Type":"ContainerDied","Data":"f13ad5b70a640cf89023d4647411ff3ddc76619ffa0b7fdc1a4eb12c0299084a"} Jul 10 19:06:29.485878 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.485853 2561 generic.go:356] "Generic (PLEG): container finished" podID="a5081e81-09af-4b03-a57e-1f1e89455699" containerID="26cf207217949bed0c249f4e9d5c928024b039d45de148c8ddd368a8f5d192d5" exitCode=0 Jul 10 19:06:29.486027 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.485882 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9" event={"ID":"a5081e81-09af-4b03-a57e-1f1e89455699","Type":"ContainerDied","Data":"26cf207217949bed0c249f4e9d5c928024b039d45de148c8ddd368a8f5d192d5"} Jul 10 19:06:29.486027 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.485958 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9" event={"ID":"a5081e81-09af-4b03-a57e-1f1e89455699","Type":"ContainerStarted","Data":"bf36cd867d68800395149bddc861382cf7fd54d3c4420c4d7210cf292a2e78cd"} Jul 10 19:06:29.521293 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.521157 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs"] Jul 10 19:06:29.526673 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.526554 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs" Jul 10 19:06:29.537433 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.537407 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs"] Jul 10 19:06:29.597081 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.597047 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd"] Jul 10 19:06:29.599540 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:06:29.599510 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8631bef5_d160_4ae4_88b0_b137c25ced1e.slice/crio-aed6e6cc5a347326fc6982f3879852adfbb8983053dd3f5c260a6746ad8de3d5 WatchSource:0}: Error finding container aed6e6cc5a347326fc6982f3879852adfbb8983053dd3f5c260a6746ad8de3d5: Status 404 returned error can't find the container with id aed6e6cc5a347326fc6982f3879852adfbb8983053dd3f5c260a6746ad8de3d5 Jul 10 19:06:29.654049 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.654019 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/855f697c-e062-4cac-88e2-9f11d5786c68-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs\" (UID: \"855f697c-e062-4cac-88e2-9f11d5786c68\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs" Jul 10 19:06:29.654238 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.654062 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28xp4\" (UniqueName: \"kubernetes.io/projected/855f697c-e062-4cac-88e2-9f11d5786c68-kube-api-access-28xp4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs\" (UID: \"855f697c-e062-4cac-88e2-9f11d5786c68\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs" Jul 10 19:06:29.654238 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.654147 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/855f697c-e062-4cac-88e2-9f11d5786c68-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs\" (UID: \"855f697c-e062-4cac-88e2-9f11d5786c68\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs" Jul 10 19:06:29.755184 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.755147 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/855f697c-e062-4cac-88e2-9f11d5786c68-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs\" (UID: \"855f697c-e062-4cac-88e2-9f11d5786c68\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs" Jul 10 19:06:29.755184 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.755186 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28xp4\" (UniqueName: \"kubernetes.io/projected/855f697c-e062-4cac-88e2-9f11d5786c68-kube-api-access-28xp4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs\" (UID: \"855f697c-e062-4cac-88e2-9f11d5786c68\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs" Jul 10 19:06:29.755394 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.755213 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/855f697c-e062-4cac-88e2-9f11d5786c68-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs\" (UID: \"855f697c-e062-4cac-88e2-9f11d5786c68\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs" Jul 10 19:06:29.755634 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.755619 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/855f697c-e062-4cac-88e2-9f11d5786c68-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs\" (UID: \"855f697c-e062-4cac-88e2-9f11d5786c68\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs" Jul 10 19:06:29.755670 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.755616 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/855f697c-e062-4cac-88e2-9f11d5786c68-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs\" (UID: \"855f697c-e062-4cac-88e2-9f11d5786c68\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs" Jul 10 19:06:29.773344 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.773312 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28xp4\" (UniqueName: \"kubernetes.io/projected/855f697c-e062-4cac-88e2-9f11d5786c68-kube-api-access-28xp4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs\" (UID: \"855f697c-e062-4cac-88e2-9f11d5786c68\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs" Jul 10 19:06:29.838444 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.838410 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs" Jul 10 19:06:29.972232 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:29.972209 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs"] Jul 10 19:06:29.974279 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:06:29.974234 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod855f697c_e062_4cac_88e2_9f11d5786c68.slice/crio-e3427c475c1046f1ca046ff2bec19d0f40db5b658c27c037f8d76a5504fb4645 WatchSource:0}: Error finding container e3427c475c1046f1ca046ff2bec19d0f40db5b658c27c037f8d76a5504fb4645: Status 404 returned error can't find the container with id e3427c475c1046f1ca046ff2bec19d0f40db5b658c27c037f8d76a5504fb4645 Jul 10 19:06:30.491744 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:30.491707 2561 generic.go:356] "Generic (PLEG): container finished" podID="8631bef5-d160-4ae4-88b0-b137c25ced1e" containerID="1c7b05d568a495ce908f986c5886706c726519ea5f0bd15691f61060db4ef32d" exitCode=0 Jul 10 19:06:30.491972 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:30.491798 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd" event={"ID":"8631bef5-d160-4ae4-88b0-b137c25ced1e","Type":"ContainerDied","Data":"1c7b05d568a495ce908f986c5886706c726519ea5f0bd15691f61060db4ef32d"} Jul 10 19:06:30.491972 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:30.491864 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd" event={"ID":"8631bef5-d160-4ae4-88b0-b137c25ced1e","Type":"ContainerStarted","Data":"aed6e6cc5a347326fc6982f3879852adfbb8983053dd3f5c260a6746ad8de3d5"} Jul 10 19:06:30.494052 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:30.494028 2561 generic.go:356] "Generic (PLEG): container finished" podID="1fb12101-ccd5-474e-b4b7-b05700be5463" containerID="decbfc97a84f49efc04f4d50d646d8e4f312751ce82ca2b8684d99a795f3f084" exitCode=0 Jul 10 19:06:30.494131 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:30.494113 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x" event={"ID":"1fb12101-ccd5-474e-b4b7-b05700be5463","Type":"ContainerDied","Data":"decbfc97a84f49efc04f4d50d646d8e4f312751ce82ca2b8684d99a795f3f084"} Jul 10 19:06:30.495896 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:30.495871 2561 generic.go:356] "Generic (PLEG): container finished" podID="a5081e81-09af-4b03-a57e-1f1e89455699" containerID="85cedf372bedcecd301cedfbebaa842548dc2609776f244a114faf773839475c" exitCode=0 Jul 10 19:06:30.496029 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:30.495896 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9" event={"ID":"a5081e81-09af-4b03-a57e-1f1e89455699","Type":"ContainerDied","Data":"85cedf372bedcecd301cedfbebaa842548dc2609776f244a114faf773839475c"} Jul 10 19:06:30.497917 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:30.497895 2561 generic.go:356] "Generic (PLEG): container finished" podID="855f697c-e062-4cac-88e2-9f11d5786c68" containerID="7015390a46bcd1c297d461e03b764185c08c174152bc90739e07b8e2d09eea0b" exitCode=0 Jul 10 19:06:30.498017 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:30.497975 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs" event={"ID":"855f697c-e062-4cac-88e2-9f11d5786c68","Type":"ContainerDied","Data":"7015390a46bcd1c297d461e03b764185c08c174152bc90739e07b8e2d09eea0b"} Jul 10 19:06:30.498071 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:30.498017 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs" event={"ID":"855f697c-e062-4cac-88e2-9f11d5786c68","Type":"ContainerStarted","Data":"e3427c475c1046f1ca046ff2bec19d0f40db5b658c27c037f8d76a5504fb4645"} Jul 10 19:06:31.504884 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:31.504854 2561 generic.go:356] "Generic (PLEG): container finished" podID="a5081e81-09af-4b03-a57e-1f1e89455699" containerID="31f1668e818cec05872a168d4173bb5a4d6e025ca062e631f73e43a3f3fc77d6" exitCode=0 Jul 10 19:06:31.505264 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:31.504922 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9" event={"ID":"a5081e81-09af-4b03-a57e-1f1e89455699","Type":"ContainerDied","Data":"31f1668e818cec05872a168d4173bb5a4d6e025ca062e631f73e43a3f3fc77d6"} Jul 10 19:06:31.506838 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:31.506812 2561 generic.go:356] "Generic (PLEG): container finished" podID="855f697c-e062-4cac-88e2-9f11d5786c68" containerID="216a66d3f069cd4cbb3c34b2869d76a58b135dff38b941905082b3c0354bfdbe" exitCode=0 Jul 10 19:06:31.506988 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:31.506851 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs" event={"ID":"855f697c-e062-4cac-88e2-9f11d5786c68","Type":"ContainerDied","Data":"216a66d3f069cd4cbb3c34b2869d76a58b135dff38b941905082b3c0354bfdbe"} Jul 10 19:06:31.708179 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:31.708143 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x" Jul 10 19:06:31.872648 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:31.872614 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmkmg\" (UniqueName: \"kubernetes.io/projected/1fb12101-ccd5-474e-b4b7-b05700be5463-kube-api-access-tmkmg\") pod \"1fb12101-ccd5-474e-b4b7-b05700be5463\" (UID: \"1fb12101-ccd5-474e-b4b7-b05700be5463\") " Jul 10 19:06:31.872850 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:31.872685 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fb12101-ccd5-474e-b4b7-b05700be5463-util\") pod \"1fb12101-ccd5-474e-b4b7-b05700be5463\" (UID: \"1fb12101-ccd5-474e-b4b7-b05700be5463\") " Jul 10 19:06:31.872850 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:31.872745 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fb12101-ccd5-474e-b4b7-b05700be5463-bundle\") pod \"1fb12101-ccd5-474e-b4b7-b05700be5463\" (UID: \"1fb12101-ccd5-474e-b4b7-b05700be5463\") " Jul 10 19:06:31.873454 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:31.873420 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb12101-ccd5-474e-b4b7-b05700be5463-bundle" (OuterVolumeSpecName: "bundle") pod "1fb12101-ccd5-474e-b4b7-b05700be5463" (UID: "1fb12101-ccd5-474e-b4b7-b05700be5463"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:06:31.875033 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:31.875010 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb12101-ccd5-474e-b4b7-b05700be5463-kube-api-access-tmkmg" (OuterVolumeSpecName: "kube-api-access-tmkmg") pod "1fb12101-ccd5-474e-b4b7-b05700be5463" (UID: "1fb12101-ccd5-474e-b4b7-b05700be5463"). InnerVolumeSpecName "kube-api-access-tmkmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:06:31.878288 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:31.878266 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb12101-ccd5-474e-b4b7-b05700be5463-util" (OuterVolumeSpecName: "util") pod "1fb12101-ccd5-474e-b4b7-b05700be5463" (UID: "1fb12101-ccd5-474e-b4b7-b05700be5463"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:06:31.974227 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:31.974201 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1fb12101-ccd5-474e-b4b7-b05700be5463-bundle\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:06:31.974227 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:31.974226 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmkmg\" (UniqueName: \"kubernetes.io/projected/1fb12101-ccd5-474e-b4b7-b05700be5463-kube-api-access-tmkmg\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:06:31.974385 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:31.974236 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1fb12101-ccd5-474e-b4b7-b05700be5463-util\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:06:32.513858 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:32.513819 2561 generic.go:356] "Generic (PLEG): container finished" podID="8631bef5-d160-4ae4-88b0-b137c25ced1e" containerID="3eb07dd3d6b333a6cd1aad63b36f37b66a31f0d375fb004576817f6d86fa786b" exitCode=0 Jul 10 19:06:32.514327 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:32.513904 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd" event={"ID":"8631bef5-d160-4ae4-88b0-b137c25ced1e","Type":"ContainerDied","Data":"3eb07dd3d6b333a6cd1aad63b36f37b66a31f0d375fb004576817f6d86fa786b"} Jul 10 19:06:32.516039 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:32.516018 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x" Jul 10 19:06:32.516147 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:32.516054 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x" event={"ID":"1fb12101-ccd5-474e-b4b7-b05700be5463","Type":"ContainerDied","Data":"cba2034726bbe391cdbf349be12e15f8cb608350cf52d68be5756735793a5ec2"} Jul 10 19:06:32.516147 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:32.516103 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cba2034726bbe391cdbf349be12e15f8cb608350cf52d68be5756735793a5ec2" Jul 10 19:06:32.518412 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:32.518391 2561 generic.go:356] "Generic (PLEG): container finished" podID="855f697c-e062-4cac-88e2-9f11d5786c68" containerID="f475c66deb73cb3b4be7729bdf63027ba352af836e7e24a614f429f9ca111268" exitCode=0 Jul 10 19:06:32.518521 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:32.518471 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs" event={"ID":"855f697c-e062-4cac-88e2-9f11d5786c68","Type":"ContainerDied","Data":"f475c66deb73cb3b4be7729bdf63027ba352af836e7e24a614f429f9ca111268"} Jul 10 19:06:32.699923 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:32.699894 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9" Jul 10 19:06:32.781599 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:32.781512 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrqfd\" (UniqueName: \"kubernetes.io/projected/a5081e81-09af-4b03-a57e-1f1e89455699-kube-api-access-mrqfd\") pod \"a5081e81-09af-4b03-a57e-1f1e89455699\" (UID: \"a5081e81-09af-4b03-a57e-1f1e89455699\") " Jul 10 19:06:32.781748 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:32.781628 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a5081e81-09af-4b03-a57e-1f1e89455699-bundle\") pod \"a5081e81-09af-4b03-a57e-1f1e89455699\" (UID: \"a5081e81-09af-4b03-a57e-1f1e89455699\") " Jul 10 19:06:32.781748 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:32.781648 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a5081e81-09af-4b03-a57e-1f1e89455699-util\") pod \"a5081e81-09af-4b03-a57e-1f1e89455699\" (UID: \"a5081e81-09af-4b03-a57e-1f1e89455699\") " Jul 10 19:06:32.782208 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:32.782173 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5081e81-09af-4b03-a57e-1f1e89455699-bundle" (OuterVolumeSpecName: "bundle") pod "a5081e81-09af-4b03-a57e-1f1e89455699" (UID: "a5081e81-09af-4b03-a57e-1f1e89455699"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:06:32.783775 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:32.783748 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5081e81-09af-4b03-a57e-1f1e89455699-kube-api-access-mrqfd" (OuterVolumeSpecName: "kube-api-access-mrqfd") pod "a5081e81-09af-4b03-a57e-1f1e89455699" (UID: "a5081e81-09af-4b03-a57e-1f1e89455699"). InnerVolumeSpecName "kube-api-access-mrqfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:06:32.787060 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:32.787038 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5081e81-09af-4b03-a57e-1f1e89455699-util" (OuterVolumeSpecName: "util") pod "a5081e81-09af-4b03-a57e-1f1e89455699" (UID: "a5081e81-09af-4b03-a57e-1f1e89455699"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:06:32.882490 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:32.882456 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a5081e81-09af-4b03-a57e-1f1e89455699-bundle\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:06:32.882490 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:32.882484 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a5081e81-09af-4b03-a57e-1f1e89455699-util\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:06:32.882490 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:32.882493 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrqfd\" (UniqueName: \"kubernetes.io/projected/a5081e81-09af-4b03-a57e-1f1e89455699-kube-api-access-mrqfd\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:06:33.525223 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:33.525193 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9" Jul 10 19:06:33.525657 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:33.525225 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9" event={"ID":"a5081e81-09af-4b03-a57e-1f1e89455699","Type":"ContainerDied","Data":"bf36cd867d68800395149bddc861382cf7fd54d3c4420c4d7210cf292a2e78cd"} Jul 10 19:06:33.525657 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:33.525279 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf36cd867d68800395149bddc861382cf7fd54d3c4420c4d7210cf292a2e78cd" Jul 10 19:06:33.527498 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:33.527471 2561 generic.go:356] "Generic (PLEG): container finished" podID="8631bef5-d160-4ae4-88b0-b137c25ced1e" containerID="4bbd56735e421618a44d32142b3206ad9c8cc599f45c69bf5e61c01c215b9cfc" exitCode=0 Jul 10 19:06:33.527618 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:33.527529 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd" event={"ID":"8631bef5-d160-4ae4-88b0-b137c25ced1e","Type":"ContainerDied","Data":"4bbd56735e421618a44d32142b3206ad9c8cc599f45c69bf5e61c01c215b9cfc"} Jul 10 19:06:33.657743 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:33.657712 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs" Jul 10 19:06:33.789713 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:33.789619 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28xp4\" (UniqueName: \"kubernetes.io/projected/855f697c-e062-4cac-88e2-9f11d5786c68-kube-api-access-28xp4\") pod \"855f697c-e062-4cac-88e2-9f11d5786c68\" (UID: \"855f697c-e062-4cac-88e2-9f11d5786c68\") " Jul 10 19:06:33.789713 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:33.789689 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/855f697c-e062-4cac-88e2-9f11d5786c68-util\") pod \"855f697c-e062-4cac-88e2-9f11d5786c68\" (UID: \"855f697c-e062-4cac-88e2-9f11d5786c68\") " Jul 10 19:06:33.789898 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:33.789742 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/855f697c-e062-4cac-88e2-9f11d5786c68-bundle\") pod \"855f697c-e062-4cac-88e2-9f11d5786c68\" (UID: \"855f697c-e062-4cac-88e2-9f11d5786c68\") " Jul 10 19:06:33.790252 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:33.790226 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855f697c-e062-4cac-88e2-9f11d5786c68-bundle" (OuterVolumeSpecName: "bundle") pod "855f697c-e062-4cac-88e2-9f11d5786c68" (UID: "855f697c-e062-4cac-88e2-9f11d5786c68"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:06:33.791951 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:33.791912 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/855f697c-e062-4cac-88e2-9f11d5786c68-kube-api-access-28xp4" (OuterVolumeSpecName: "kube-api-access-28xp4") pod "855f697c-e062-4cac-88e2-9f11d5786c68" (UID: "855f697c-e062-4cac-88e2-9f11d5786c68"). InnerVolumeSpecName "kube-api-access-28xp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:06:33.795139 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:33.795103 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855f697c-e062-4cac-88e2-9f11d5786c68-util" (OuterVolumeSpecName: "util") pod "855f697c-e062-4cac-88e2-9f11d5786c68" (UID: "855f697c-e062-4cac-88e2-9f11d5786c68"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:06:33.890531 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:33.890496 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28xp4\" (UniqueName: \"kubernetes.io/projected/855f697c-e062-4cac-88e2-9f11d5786c68-kube-api-access-28xp4\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:06:33.890531 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:33.890524 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/855f697c-e062-4cac-88e2-9f11d5786c68-util\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:06:33.890531 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:33.890537 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/855f697c-e062-4cac-88e2-9f11d5786c68-bundle\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:06:34.534045 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:34.534009 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs" Jul 10 19:06:34.534045 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:34.534036 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs" event={"ID":"855f697c-e062-4cac-88e2-9f11d5786c68","Type":"ContainerDied","Data":"e3427c475c1046f1ca046ff2bec19d0f40db5b658c27c037f8d76a5504fb4645"} Jul 10 19:06:34.534444 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:34.534125 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3427c475c1046f1ca046ff2bec19d0f40db5b658c27c037f8d76a5504fb4645" Jul 10 19:06:34.666646 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:34.666614 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd" Jul 10 19:06:34.798214 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:34.798123 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8631bef5-d160-4ae4-88b0-b137c25ced1e-bundle\") pod \"8631bef5-d160-4ae4-88b0-b137c25ced1e\" (UID: \"8631bef5-d160-4ae4-88b0-b137c25ced1e\") " Jul 10 19:06:34.798214 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:34.798192 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8631bef5-d160-4ae4-88b0-b137c25ced1e-util\") pod \"8631bef5-d160-4ae4-88b0-b137c25ced1e\" (UID: \"8631bef5-d160-4ae4-88b0-b137c25ced1e\") " Jul 10 19:06:34.798442 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:34.798242 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rcvn\" (UniqueName: \"kubernetes.io/projected/8631bef5-d160-4ae4-88b0-b137c25ced1e-kube-api-access-4rcvn\") pod \"8631bef5-d160-4ae4-88b0-b137c25ced1e\" (UID: \"8631bef5-d160-4ae4-88b0-b137c25ced1e\") " Jul 10 19:06:34.798555 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:34.798530 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8631bef5-d160-4ae4-88b0-b137c25ced1e-bundle" (OuterVolumeSpecName: "bundle") pod "8631bef5-d160-4ae4-88b0-b137c25ced1e" (UID: "8631bef5-d160-4ae4-88b0-b137c25ced1e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:06:34.800483 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:34.800459 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8631bef5-d160-4ae4-88b0-b137c25ced1e-kube-api-access-4rcvn" (OuterVolumeSpecName: "kube-api-access-4rcvn") pod "8631bef5-d160-4ae4-88b0-b137c25ced1e" (UID: "8631bef5-d160-4ae4-88b0-b137c25ced1e"). InnerVolumeSpecName "kube-api-access-4rcvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:06:34.803828 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:34.803803 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8631bef5-d160-4ae4-88b0-b137c25ced1e-util" (OuterVolumeSpecName: "util") pod "8631bef5-d160-4ae4-88b0-b137c25ced1e" (UID: "8631bef5-d160-4ae4-88b0-b137c25ced1e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:06:34.899082 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:34.899029 2561 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8631bef5-d160-4ae4-88b0-b137c25ced1e-bundle\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:06:34.899082 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:34.899074 2561 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8631bef5-d160-4ae4-88b0-b137c25ced1e-util\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:06:34.899082 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:34.899086 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rcvn\" (UniqueName: \"kubernetes.io/projected/8631bef5-d160-4ae4-88b0-b137c25ced1e-kube-api-access-4rcvn\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:06:35.541106 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:35.541074 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd" Jul 10 19:06:35.541106 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:35.541102 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd" event={"ID":"8631bef5-d160-4ae4-88b0-b137c25ced1e","Type":"ContainerDied","Data":"aed6e6cc5a347326fc6982f3879852adfbb8983053dd3f5c260a6746ad8de3d5"} Jul 10 19:06:35.541522 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:35.541146 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aed6e6cc5a347326fc6982f3879852adfbb8983053dd3f5c260a6746ad8de3d5" Jul 10 19:06:40.983368 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:40.983338 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:41.578674 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:41.578639 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:42.589114 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:42.589079 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:46.196768 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:46.196738 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:48.577843 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:48.577802 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:48.797896 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:48.797858 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:51.025952 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:51.025901 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:51.162376 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:51.162342 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:53.664565 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:53.664538 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:56.298927 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:56.298893 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:56.424026 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:56.423995 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:56.525527 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:56.525482 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:06:59.993378 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:06:59.993349 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:00.096213 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:00.096149 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:01.241566 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241530 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8wrx4"] Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241814 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8631bef5-d160-4ae4-88b0-b137c25ced1e" containerName="util" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241825 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8631bef5-d160-4ae4-88b0-b137c25ced1e" containerName="util" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241835 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fb12101-ccd5-474e-b4b7-b05700be5463" containerName="util" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241840 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fb12101-ccd5-474e-b4b7-b05700be5463" containerName="util" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241847 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="855f697c-e062-4cac-88e2-9f11d5786c68" containerName="util" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241853 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="855f697c-e062-4cac-88e2-9f11d5786c68" containerName="util" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241859 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8631bef5-d160-4ae4-88b0-b137c25ced1e" containerName="pull" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241863 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8631bef5-d160-4ae4-88b0-b137c25ced1e" containerName="pull" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241872 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5081e81-09af-4b03-a57e-1f1e89455699" containerName="util" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241877 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5081e81-09af-4b03-a57e-1f1e89455699" containerName="util" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241883 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fb12101-ccd5-474e-b4b7-b05700be5463" containerName="extract" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241888 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fb12101-ccd5-474e-b4b7-b05700be5463" containerName="extract" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241894 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5081e81-09af-4b03-a57e-1f1e89455699" containerName="extract" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241900 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5081e81-09af-4b03-a57e-1f1e89455699" containerName="extract" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241911 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5081e81-09af-4b03-a57e-1f1e89455699" containerName="pull" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241916 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5081e81-09af-4b03-a57e-1f1e89455699" containerName="pull" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241921 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="855f697c-e062-4cac-88e2-9f11d5786c68" containerName="extract" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241927 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="855f697c-e062-4cac-88e2-9f11d5786c68" containerName="extract" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241949 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="855f697c-e062-4cac-88e2-9f11d5786c68" containerName="pull" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241954 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="855f697c-e062-4cac-88e2-9f11d5786c68" containerName="pull" Jul 10 19:07:01.241950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241960 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fb12101-ccd5-474e-b4b7-b05700be5463" containerName="pull" Jul 10 19:07:01.242593 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241964 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fb12101-ccd5-474e-b4b7-b05700be5463" containerName="pull" Jul 10 19:07:01.242593 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241971 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8631bef5-d160-4ae4-88b0-b137c25ced1e" containerName="extract" Jul 10 19:07:01.242593 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.241976 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8631bef5-d160-4ae4-88b0-b137c25ced1e" containerName="extract" Jul 10 19:07:01.242593 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.242024 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5081e81-09af-4b03-a57e-1f1e89455699" containerName="extract" Jul 10 19:07:01.242593 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.242033 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fb12101-ccd5-474e-b4b7-b05700be5463" containerName="extract" Jul 10 19:07:01.242593 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.242039 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8631bef5-d160-4ae4-88b0-b137c25ced1e" containerName="extract" Jul 10 19:07:01.242593 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.242046 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="855f697c-e062-4cac-88e2-9f11d5786c68" containerName="extract" Jul 10 19:07:01.244869 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.244853 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8wrx4" Jul 10 19:07:01.247703 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.247672 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 19:07:01.247840 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.247753 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 10 19:07:01.247929 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.247907 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 10 19:07:01.249636 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.249618 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 19:07:01.249860 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.249844 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-k59z6\"" Jul 10 19:07:01.260585 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.260550 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:01.261503 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.261483 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8wrx4"] Jul 10 19:07:01.302211 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.302178 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e249aad8-34ff-4221-876e-2d907faa19f3-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8wrx4\" (UID: \"e249aad8-34ff-4221-876e-2d907faa19f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8wrx4" Jul 10 19:07:01.302211 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.302209 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbnx2\" (UniqueName: \"kubernetes.io/projected/e249aad8-34ff-4221-876e-2d907faa19f3-kube-api-access-xbnx2\") pod \"kuadrant-console-plugin-66b5774985-8wrx4\" (UID: \"e249aad8-34ff-4221-876e-2d907faa19f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8wrx4" Jul 10 19:07:01.302438 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.302256 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e249aad8-34ff-4221-876e-2d907faa19f3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8wrx4\" (UID: \"e249aad8-34ff-4221-876e-2d907faa19f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8wrx4" Jul 10 19:07:01.370972 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.370910 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:01.403229 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.403193 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e249aad8-34ff-4221-876e-2d907faa19f3-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8wrx4\" (UID: \"e249aad8-34ff-4221-876e-2d907faa19f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8wrx4" Jul 10 19:07:01.403397 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.403234 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbnx2\" (UniqueName: \"kubernetes.io/projected/e249aad8-34ff-4221-876e-2d907faa19f3-kube-api-access-xbnx2\") pod \"kuadrant-console-plugin-66b5774985-8wrx4\" (UID: \"e249aad8-34ff-4221-876e-2d907faa19f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8wrx4" Jul 10 19:07:01.403397 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.403304 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e249aad8-34ff-4221-876e-2d907faa19f3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8wrx4\" (UID: \"e249aad8-34ff-4221-876e-2d907faa19f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8wrx4" Jul 10 19:07:01.403931 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.403912 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e249aad8-34ff-4221-876e-2d907faa19f3-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-8wrx4\" (UID: \"e249aad8-34ff-4221-876e-2d907faa19f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8wrx4" Jul 10 19:07:01.405904 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.405883 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e249aad8-34ff-4221-876e-2d907faa19f3-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-8wrx4\" (UID: \"e249aad8-34ff-4221-876e-2d907faa19f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8wrx4" Jul 10 19:07:01.413483 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.413458 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbnx2\" (UniqueName: \"kubernetes.io/projected/e249aad8-34ff-4221-876e-2d907faa19f3-kube-api-access-xbnx2\") pod \"kuadrant-console-plugin-66b5774985-8wrx4\" (UID: \"e249aad8-34ff-4221-876e-2d907faa19f3\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8wrx4" Jul 10 19:07:01.553839 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.553810 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8wrx4" Jul 10 19:07:01.695010 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:01.694983 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-8wrx4"] Jul 10 19:07:01.696930 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:07:01.696901 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode249aad8_34ff_4221_876e_2d907faa19f3.slice/crio-06377ab87b249865f52b7bed39c5a04cf5750cbbbef53568ac1833de512ee39c WatchSource:0}: Error finding container 06377ab87b249865f52b7bed39c5a04cf5750cbbbef53568ac1833de512ee39c: Status 404 returned error can't find the container with id 06377ab87b249865f52b7bed39c5a04cf5750cbbbef53568ac1833de512ee39c Jul 10 19:07:02.090651 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:02.090619 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:02.196548 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:02.196513 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:02.667605 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:02.667565 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8wrx4" event={"ID":"e249aad8-34ff-4221-876e-2d907faa19f3","Type":"ContainerStarted","Data":"06377ab87b249865f52b7bed39c5a04cf5750cbbbef53568ac1833de512ee39c"} Jul 10 19:07:04.052148 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:04.052111 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:04.167650 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:04.167619 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:04.310538 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:04.310410 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:07.495891 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:07.495854 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:07.594431 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:07.594396 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:15.597506 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:15.597465 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:15.684036 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:15.684001 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:16.185966 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:16.185913 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:16.263488 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:16.263455 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:18.934614 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:18.934573 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:19.030122 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:19.030086 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:19.098679 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:19.098643 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:19.182649 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:19.182606 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:19.270526 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:19.270494 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:23.814445 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:23.814404 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:23.908241 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:23.908205 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:24.790563 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:24.790520 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8wrx4" event={"ID":"e249aad8-34ff-4221-876e-2d907faa19f3","Type":"ContainerStarted","Data":"c75f7b1a6c91de38a6e9b5a9c7e136f6e36663c2c92cbcfd79c05c0d03f37391"} Jul 10 19:07:24.815873 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:24.815814 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-8wrx4" podStartSLOduration=1.265376735 podStartE2EDuration="23.815792635s" totalImagesPullingTime="22.5504159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:07:24.814310397 +0000 UTC m=+411.415452512" watchObservedRunningTime="2026-07-10 19:07:24.815792635 +0000 UTC m=+411.416934771" Jul 10 19:07:26.330494 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:26.330466 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:26.430650 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:26.430618 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:40.547868 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:40.547829 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:40.885583 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:40.885481 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:55.845652 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:55.845559 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:55.949989 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:55.949956 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:56.052673 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:56.052639 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:56.268412 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:56.268376 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:07:56.281573 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:56.281539 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:07:56.281732 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:56.281629 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-4kfc2" Jul 10 19:07:56.283789 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:56.283767 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 19:07:56.320819 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:56.320783 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:07:56.361129 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:56.361093 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45700ba9-9c1d-4475-9267-6a89879dc36e-config-file\") pod \"limitador-limitador-54d6d988d5-4kfc2\" (UID: \"45700ba9-9c1d-4475-9267-6a89879dc36e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4kfc2" Jul 10 19:07:56.361295 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:56.361142 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh76n\" (UniqueName: \"kubernetes.io/projected/45700ba9-9c1d-4475-9267-6a89879dc36e-kube-api-access-mh76n\") pod \"limitador-limitador-54d6d988d5-4kfc2\" (UID: \"45700ba9-9c1d-4475-9267-6a89879dc36e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4kfc2" Jul 10 19:07:56.462234 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:56.462198 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh76n\" (UniqueName: \"kubernetes.io/projected/45700ba9-9c1d-4475-9267-6a89879dc36e-kube-api-access-mh76n\") pod \"limitador-limitador-54d6d988d5-4kfc2\" (UID: \"45700ba9-9c1d-4475-9267-6a89879dc36e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4kfc2" Jul 10 19:07:56.462371 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:56.462292 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45700ba9-9c1d-4475-9267-6a89879dc36e-config-file\") pod \"limitador-limitador-54d6d988d5-4kfc2\" (UID: \"45700ba9-9c1d-4475-9267-6a89879dc36e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4kfc2" Jul 10 19:07:56.462989 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:56.462917 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/45700ba9-9c1d-4475-9267-6a89879dc36e-config-file\") pod \"limitador-limitador-54d6d988d5-4kfc2\" (UID: \"45700ba9-9c1d-4475-9267-6a89879dc36e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4kfc2" Jul 10 19:07:56.472060 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:56.472038 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh76n\" (UniqueName: \"kubernetes.io/projected/45700ba9-9c1d-4475-9267-6a89879dc36e-kube-api-access-mh76n\") pod \"limitador-limitador-54d6d988d5-4kfc2\" (UID: \"45700ba9-9c1d-4475-9267-6a89879dc36e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-4kfc2" Jul 10 19:07:56.592531 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:56.592438 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-4kfc2" Jul 10 19:07:56.683104 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:56.683068 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:56.728639 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:56.728609 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:07:56.731318 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:07:56.731290 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45700ba9_9c1d_4475_9267_6a89879dc36e.slice/crio-05869d58eb1bac00a73fff6b229670bc85fa7bd79e2616f48db56b9cc091e96f WatchSource:0}: Error finding container 05869d58eb1bac00a73fff6b229670bc85fa7bd79e2616f48db56b9cc091e96f: Status 404 returned error can't find the container with id 05869d58eb1bac00a73fff6b229670bc85fa7bd79e2616f48db56b9cc091e96f Jul 10 19:07:56.740471 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:56.740445 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:07:56.939882 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:56.939788 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-4kfc2" event={"ID":"45700ba9-9c1d-4475-9267-6a89879dc36e","Type":"ContainerStarted","Data":"05869d58eb1bac00a73fff6b229670bc85fa7bd79e2616f48db56b9cc091e96f"} Jul 10 19:07:59.962031 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:59.962004 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-4kfc2" Jul 10 19:07:59.962408 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:59.962051 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-4kfc2" event={"ID":"45700ba9-9c1d-4475-9267-6a89879dc36e","Type":"ContainerStarted","Data":"ea0aabd5f9d7c7be40e998d4f092102cf9e76225d91253ab7007f625b9460a02"} Jul 10 19:07:59.979888 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:07:59.979833 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-4kfc2" podStartSLOduration=1.575591598 podStartE2EDuration="3.979816272s" totalImagesPullingTime="2.404224674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:07:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:07:59.978090142 +0000 UTC m=+446.579232278" watchObservedRunningTime="2026-07-10 19:07:59.979816272 +0000 UTC m=+446.580958407" Jul 10 19:08:10.963832 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:10.963801 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-4kfc2" Jul 10 19:08:13.656200 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:13.656168 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:13.763163 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:13.763126 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:14.175541 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:14.175504 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:26.828424 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:26.828380 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:27.192530 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:27.192439 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:27.313664 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:27.313631 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76f48c9dc4-d4rpv"] Jul 10 19:08:27.316789 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:27.316773 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76f48c9dc4-d4rpv" Jul 10 19:08:27.321041 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:27.321019 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jdvs4\"" Jul 10 19:08:27.329419 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:27.329393 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76f48c9dc4-d4rpv"] Jul 10 19:08:27.406106 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:27.406066 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj9st\" (UniqueName: \"kubernetes.io/projected/3f48197e-4757-47d6-aefc-dab80a9ce3cd-kube-api-access-nj9st\") pod \"authorino-76f48c9dc4-d4rpv\" (UID: \"3f48197e-4757-47d6-aefc-dab80a9ce3cd\") " pod="kuadrant-system/authorino-76f48c9dc4-d4rpv" Jul 10 19:08:27.506882 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:27.506847 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj9st\" (UniqueName: \"kubernetes.io/projected/3f48197e-4757-47d6-aefc-dab80a9ce3cd-kube-api-access-nj9st\") pod \"authorino-76f48c9dc4-d4rpv\" (UID: \"3f48197e-4757-47d6-aefc-dab80a9ce3cd\") " pod="kuadrant-system/authorino-76f48c9dc4-d4rpv" Jul 10 19:08:27.518794 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:27.518763 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj9st\" (UniqueName: \"kubernetes.io/projected/3f48197e-4757-47d6-aefc-dab80a9ce3cd-kube-api-access-nj9st\") pod \"authorino-76f48c9dc4-d4rpv\" (UID: \"3f48197e-4757-47d6-aefc-dab80a9ce3cd\") " pod="kuadrant-system/authorino-76f48c9dc4-d4rpv" Jul 10 19:08:27.540774 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:27.540738 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76f48c9dc4-d4rpv"] Jul 10 19:08:27.545427 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:27.545401 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76f48c9dc4-d4rpv" Jul 10 19:08:27.678724 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:27.678700 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76f48c9dc4-d4rpv"] Jul 10 19:08:27.680463 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:08:27.680435 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f48197e_4757_47d6_aefc_dab80a9ce3cd.slice/crio-4eb702c6128436200bf4647baca6c1ba1ad2280221bacd0e05c21e65a395f769 WatchSource:0}: Error finding container 4eb702c6128436200bf4647baca6c1ba1ad2280221bacd0e05c21e65a395f769: Status 404 returned error can't find the container with id 4eb702c6128436200bf4647baca6c1ba1ad2280221bacd0e05c21e65a395f769 Jul 10 19:08:28.083189 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:28.083151 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76f48c9dc4-d4rpv" event={"ID":"3f48197e-4757-47d6-aefc-dab80a9ce3cd","Type":"ContainerStarted","Data":"4eb702c6128436200bf4647baca6c1ba1ad2280221bacd0e05c21e65a395f769"} Jul 10 19:08:32.105641 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:32.105598 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76f48c9dc4-d4rpv" event={"ID":"3f48197e-4757-47d6-aefc-dab80a9ce3cd","Type":"ContainerStarted","Data":"7971471837cdfc8195b3eb8fabeaf5d0f5a506d41fa4d09e2c26468ff76b68af"} Jul 10 19:08:32.105641 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:32.105607 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-76f48c9dc4-d4rpv" podUID="3f48197e-4757-47d6-aefc-dab80a9ce3cd" containerName="authorino" containerID="cri-o://7971471837cdfc8195b3eb8fabeaf5d0f5a506d41fa4d09e2c26468ff76b68af" gracePeriod=30 Jul 10 19:08:32.125164 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:32.125111 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-76f48c9dc4-d4rpv" podStartSLOduration=1.760000809 podStartE2EDuration="5.125092991s" totalImagesPullingTime="3.365092182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:08:32.123056758 +0000 UTC m=+478.724198896" watchObservedRunningTime="2026-07-10 19:08:32.125092991 +0000 UTC m=+478.726235126" Jul 10 19:08:32.357978 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:32.357897 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76f48c9dc4-d4rpv" Jul 10 19:08:32.449277 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:32.449241 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nj9st\" (UniqueName: \"kubernetes.io/projected/3f48197e-4757-47d6-aefc-dab80a9ce3cd-kube-api-access-nj9st\") pod \"3f48197e-4757-47d6-aefc-dab80a9ce3cd\" (UID: \"3f48197e-4757-47d6-aefc-dab80a9ce3cd\") " Jul 10 19:08:32.451415 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:32.451385 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f48197e-4757-47d6-aefc-dab80a9ce3cd-kube-api-access-nj9st" (OuterVolumeSpecName: "kube-api-access-nj9st") pod "3f48197e-4757-47d6-aefc-dab80a9ce3cd" (UID: "3f48197e-4757-47d6-aefc-dab80a9ce3cd"). InnerVolumeSpecName "kube-api-access-nj9st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:08:32.550592 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:32.550540 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj9st\" (UniqueName: \"kubernetes.io/projected/3f48197e-4757-47d6-aefc-dab80a9ce3cd-kube-api-access-nj9st\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:08:33.111714 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:33.111674 2561 generic.go:356] "Generic (PLEG): container finished" podID="3f48197e-4757-47d6-aefc-dab80a9ce3cd" containerID="7971471837cdfc8195b3eb8fabeaf5d0f5a506d41fa4d09e2c26468ff76b68af" exitCode=0 Jul 10 19:08:33.112117 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:33.111731 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76f48c9dc4-d4rpv" Jul 10 19:08:33.112117 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:33.111755 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76f48c9dc4-d4rpv" event={"ID":"3f48197e-4757-47d6-aefc-dab80a9ce3cd","Type":"ContainerDied","Data":"7971471837cdfc8195b3eb8fabeaf5d0f5a506d41fa4d09e2c26468ff76b68af"} Jul 10 19:08:33.112117 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:33.111797 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76f48c9dc4-d4rpv" event={"ID":"3f48197e-4757-47d6-aefc-dab80a9ce3cd","Type":"ContainerDied","Data":"4eb702c6128436200bf4647baca6c1ba1ad2280221bacd0e05c21e65a395f769"} Jul 10 19:08:33.112117 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:33.111814 2561 scope.go:117] "RemoveContainer" containerID="7971471837cdfc8195b3eb8fabeaf5d0f5a506d41fa4d09e2c26468ff76b68af" Jul 10 19:08:33.120782 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:33.120765 2561 scope.go:117] "RemoveContainer" containerID="7971471837cdfc8195b3eb8fabeaf5d0f5a506d41fa4d09e2c26468ff76b68af" Jul 10 19:08:33.121133 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:08:33.121111 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7971471837cdfc8195b3eb8fabeaf5d0f5a506d41fa4d09e2c26468ff76b68af\": container with ID starting with 7971471837cdfc8195b3eb8fabeaf5d0f5a506d41fa4d09e2c26468ff76b68af not found: ID does not exist" containerID="7971471837cdfc8195b3eb8fabeaf5d0f5a506d41fa4d09e2c26468ff76b68af" Jul 10 19:08:33.121186 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:33.121142 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7971471837cdfc8195b3eb8fabeaf5d0f5a506d41fa4d09e2c26468ff76b68af"} err="failed to get container status \"7971471837cdfc8195b3eb8fabeaf5d0f5a506d41fa4d09e2c26468ff76b68af\": rpc error: code = NotFound desc = could not find container \"7971471837cdfc8195b3eb8fabeaf5d0f5a506d41fa4d09e2c26468ff76b68af\": container with ID starting with 7971471837cdfc8195b3eb8fabeaf5d0f5a506d41fa4d09e2c26468ff76b68af not found: ID does not exist" Jul 10 19:08:33.137864 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:33.137827 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76f48c9dc4-d4rpv"] Jul 10 19:08:33.142143 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:33.142123 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76f48c9dc4-d4rpv"] Jul 10 19:08:33.963372 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:33.963340 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f48197e-4757-47d6-aefc-dab80a9ce3cd" path="/var/lib/kubelet/pods/3f48197e-4757-47d6-aefc-dab80a9ce3cd/volumes" Jul 10 19:08:34.087182 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:34.087149 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:34.185270 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:34.185176 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:34.286777 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:34.286738 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:34.380779 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:34.380740 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:37.152661 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:37.152629 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:08:37.260607 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:08:37.260573 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:00.885659 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:09:00.885620 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:08.637949 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:09:08.637897 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:09.115503 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:09:09.115473 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:09.438201 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:09:09.438112 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:09.675338 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:09:09.675302 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:09.774918 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:09:09.774867 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:09.803769 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:09:09.803735 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:09:20.020640 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:09:20.020607 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:20.130872 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:09:20.130838 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:30.736255 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:09:30.736219 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:31.395461 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:09:31.395418 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:09:36.895741 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:09:36.895707 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:37.601977 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:09:37.601922 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:09:52.329736 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:09:52.329700 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:09:53.694197 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:09:53.694162 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:10:10.803751 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:10.803713 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:10:12.300649 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:12.300612 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:10:23.667877 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:23.667844 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:10:25.170795 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:25.170760 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:10:26.795602 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:26.795563 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:10:27.241749 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:27.241666 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:10:33.875205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:33.875176 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:10:33.877328 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:33.877307 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:10:46.312165 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.312125 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm"] Jul 10 19:10:46.314395 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.312537 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f48197e-4757-47d6-aefc-dab80a9ce3cd" containerName="authorino" Jul 10 19:10:46.314395 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.312552 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f48197e-4757-47d6-aefc-dab80a9ce3cd" containerName="authorino" Jul 10 19:10:46.314395 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.312604 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f48197e-4757-47d6-aefc-dab80a9ce3cd" containerName="authorino" Jul 10 19:10:46.315267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.315253 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.319673 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.319648 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 19:10:46.319806 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.319695 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 19:10:46.319806 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.319707 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 10 19:10:46.319806 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.319693 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vc5dn\"" Jul 10 19:10:46.331636 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.331601 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm"] Jul 10 19:10:46.332060 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.332042 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:10:46.402261 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.402222 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cec515a-619d-4d1a-b7d8-8505938a988f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm\" (UID: \"7cec515a-619d-4d1a-b7d8-8505938a988f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.402261 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.402264 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cec515a-619d-4d1a-b7d8-8505938a988f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm\" (UID: \"7cec515a-619d-4d1a-b7d8-8505938a988f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.402506 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.402296 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cec515a-619d-4d1a-b7d8-8505938a988f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm\" (UID: \"7cec515a-619d-4d1a-b7d8-8505938a988f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.402506 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.402351 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phctq\" (UniqueName: \"kubernetes.io/projected/7cec515a-619d-4d1a-b7d8-8505938a988f-kube-api-access-phctq\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm\" (UID: \"7cec515a-619d-4d1a-b7d8-8505938a988f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.402506 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.402380 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cec515a-619d-4d1a-b7d8-8505938a988f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm\" (UID: \"7cec515a-619d-4d1a-b7d8-8505938a988f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.402506 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.402427 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cec515a-619d-4d1a-b7d8-8505938a988f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm\" (UID: \"7cec515a-619d-4d1a-b7d8-8505938a988f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.503587 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.503550 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cec515a-619d-4d1a-b7d8-8505938a988f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm\" (UID: \"7cec515a-619d-4d1a-b7d8-8505938a988f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.503765 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.503598 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cec515a-619d-4d1a-b7d8-8505938a988f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm\" (UID: \"7cec515a-619d-4d1a-b7d8-8505938a988f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.503765 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.503624 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cec515a-619d-4d1a-b7d8-8505938a988f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm\" (UID: \"7cec515a-619d-4d1a-b7d8-8505938a988f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.503765 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.503661 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cec515a-619d-4d1a-b7d8-8505938a988f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm\" (UID: \"7cec515a-619d-4d1a-b7d8-8505938a988f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.503765 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.503684 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phctq\" (UniqueName: \"kubernetes.io/projected/7cec515a-619d-4d1a-b7d8-8505938a988f-kube-api-access-phctq\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm\" (UID: \"7cec515a-619d-4d1a-b7d8-8505938a988f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.503765 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.503699 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cec515a-619d-4d1a-b7d8-8505938a988f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm\" (UID: \"7cec515a-619d-4d1a-b7d8-8505938a988f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.504105 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.504082 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cec515a-619d-4d1a-b7d8-8505938a988f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm\" (UID: \"7cec515a-619d-4d1a-b7d8-8505938a988f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.504160 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.504099 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7cec515a-619d-4d1a-b7d8-8505938a988f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm\" (UID: \"7cec515a-619d-4d1a-b7d8-8505938a988f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.504199 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.504155 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7cec515a-619d-4d1a-b7d8-8505938a988f-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm\" (UID: \"7cec515a-619d-4d1a-b7d8-8505938a988f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.506088 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.506069 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7cec515a-619d-4d1a-b7d8-8505938a988f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm\" (UID: \"7cec515a-619d-4d1a-b7d8-8505938a988f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.506361 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.506342 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7cec515a-619d-4d1a-b7d8-8505938a988f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm\" (UID: \"7cec515a-619d-4d1a-b7d8-8505938a988f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.514306 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.514277 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phctq\" (UniqueName: \"kubernetes.io/projected/7cec515a-619d-4d1a-b7d8-8505938a988f-kube-api-access-phctq\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm\" (UID: \"7cec515a-619d-4d1a-b7d8-8505938a988f\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.624187 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.624086 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:46.761607 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:46.761576 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm"] Jul 10 19:10:46.763164 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:10:46.763130 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cec515a_619d_4d1a_b7d8_8505938a988f.slice/crio-9fe4372b6feeb8fa98d1b27c257fdc9f3c895010344fd8688cc3a104276a9551 WatchSource:0}: Error finding container 9fe4372b6feeb8fa98d1b27c257fdc9f3c895010344fd8688cc3a104276a9551: Status 404 returned error can't find the container with id 9fe4372b6feeb8fa98d1b27c257fdc9f3c895010344fd8688cc3a104276a9551 Jul 10 19:10:47.772267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:47.772227 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" event={"ID":"7cec515a-619d-4d1a-b7d8-8505938a988f","Type":"ContainerStarted","Data":"9fe4372b6feeb8fa98d1b27c257fdc9f3c895010344fd8688cc3a104276a9551"} Jul 10 19:10:48.702978 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:48.702927 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:10:48.778931 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:48.778889 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" event={"ID":"7cec515a-619d-4d1a-b7d8-8505938a988f","Type":"ContainerStarted","Data":"f680a5b5b01c91b528505ac69215e875a86dfb944f2deed946707bcd2958cf1b"} Jul 10 19:10:48.801232 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:48.801178 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" podStartSLOduration=1.76426762 podStartE2EDuration="2.801159639s" totalImagesPullingTime="1.036892019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:10:48.800474724 +0000 UTC m=+615.401616860" watchObservedRunningTime="2026-07-10 19:10:48.801159639 +0000 UTC m=+615.402301773" Jul 10 19:10:56.624840 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:56.624750 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:56.624840 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:56.624799 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:56.637430 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:56.637404 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:10:56.827051 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:10:56.827018 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm" Jul 10 19:11:00.933480 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:00.933445 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:01.033573 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:01.033543 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:22.068679 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:22.068640 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:45.313731 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:45.313695 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bbc49449f-snpzm"] Jul 10 19:11:45.316504 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:45.316488 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bbc49449f-snpzm" Jul 10 19:11:45.320340 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:45.320321 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jdvs4\"" Jul 10 19:11:45.321278 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:45.321253 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 10 19:11:45.327639 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:45.327619 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bbc49449f-snpzm"] Jul 10 19:11:45.390322 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:45.390288 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc4jc\" (UniqueName: \"kubernetes.io/projected/44e764e6-1713-43ea-b2c6-a90c293a8d66-kube-api-access-nc4jc\") pod \"authorino-bbc49449f-snpzm\" (UID: \"44e764e6-1713-43ea-b2c6-a90c293a8d66\") " pod="kuadrant-system/authorino-bbc49449f-snpzm" Jul 10 19:11:45.390490 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:45.390352 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44e764e6-1713-43ea-b2c6-a90c293a8d66-tls-cert\") pod \"authorino-bbc49449f-snpzm\" (UID: \"44e764e6-1713-43ea-b2c6-a90c293a8d66\") " pod="kuadrant-system/authorino-bbc49449f-snpzm" Jul 10 19:11:45.491625 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:45.491585 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44e764e6-1713-43ea-b2c6-a90c293a8d66-tls-cert\") pod \"authorino-bbc49449f-snpzm\" (UID: \"44e764e6-1713-43ea-b2c6-a90c293a8d66\") " pod="kuadrant-system/authorino-bbc49449f-snpzm" Jul 10 19:11:45.491789 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:45.491647 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc4jc\" (UniqueName: \"kubernetes.io/projected/44e764e6-1713-43ea-b2c6-a90c293a8d66-kube-api-access-nc4jc\") pod \"authorino-bbc49449f-snpzm\" (UID: \"44e764e6-1713-43ea-b2c6-a90c293a8d66\") " pod="kuadrant-system/authorino-bbc49449f-snpzm" Jul 10 19:11:45.494220 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:45.494190 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/44e764e6-1713-43ea-b2c6-a90c293a8d66-tls-cert\") pod \"authorino-bbc49449f-snpzm\" (UID: \"44e764e6-1713-43ea-b2c6-a90c293a8d66\") " pod="kuadrant-system/authorino-bbc49449f-snpzm" Jul 10 19:11:45.501738 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:45.501715 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc4jc\" (UniqueName: \"kubernetes.io/projected/44e764e6-1713-43ea-b2c6-a90c293a8d66-kube-api-access-nc4jc\") pod \"authorino-bbc49449f-snpzm\" (UID: \"44e764e6-1713-43ea-b2c6-a90c293a8d66\") " pod="kuadrant-system/authorino-bbc49449f-snpzm" Jul 10 19:11:45.626572 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:45.626501 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bbc49449f-snpzm" Jul 10 19:11:45.694747 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:45.694720 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:45.763856 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:11:45.763828 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44e764e6_1713_43ea_b2c6_a90c293a8d66.slice/crio-1357c33261c852ff367fbb5b856d2f19d21949be6f0c22326ecbedd1989903bd WatchSource:0}: Error finding container 1357c33261c852ff367fbb5b856d2f19d21949be6f0c22326ecbedd1989903bd: Status 404 returned error can't find the container with id 1357c33261c852ff367fbb5b856d2f19d21949be6f0c22326ecbedd1989903bd Jul 10 19:11:45.765178 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:45.765159 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:11:45.765290 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:45.765269 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bbc49449f-snpzm"] Jul 10 19:11:45.780778 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:45.780749 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:46.046904 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:46.046864 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bbc49449f-snpzm" event={"ID":"44e764e6-1713-43ea-b2c6-a90c293a8d66","Type":"ContainerStarted","Data":"1357c33261c852ff367fbb5b856d2f19d21949be6f0c22326ecbedd1989903bd"} Jul 10 19:11:46.829508 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:46.829458 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:46.976479 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:46.976433 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:11:47.055784 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:47.055729 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bbc49449f-snpzm" event={"ID":"44e764e6-1713-43ea-b2c6-a90c293a8d66","Type":"ContainerStarted","Data":"132be0565eaffeb31b1f61dec0d43839e1fe4b36844b06eb46c3cd7dbeaaa086"} Jul 10 19:11:47.081547 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:11:47.081451 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-bbc49449f-snpzm" podStartSLOduration=1.6391576030000001 podStartE2EDuration="2.081436195s" totalImagesPullingTime="442.278592ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:11:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:11:47.078188277 +0000 UTC m=+673.679330412" watchObservedRunningTime="2026-07-10 19:11:47.081436195 +0000 UTC m=+673.682578331" Jul 10 19:12:13.914710 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:12:13.914666 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:12:23.503638 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:12:23.503599 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:12:36.003136 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:12:36.003099 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:12:44.210209 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:12:44.210174 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:12:54.392126 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:12:54.392086 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:12:56.947396 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:12:56.947347 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:12:58.110073 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:12:58.110039 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:05.321667 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:13:05.321634 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:21.568011 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:13:21.567968 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:13:34.195198 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:13:34.195158 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:13:38.592820 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:13:38.592786 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:38.688963 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:13:38.688910 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:38.837090 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:13:38.837058 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:45.266261 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:13:45.266229 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:45.329352 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:13:45.329317 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:45.426606 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:13:45.426573 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:45.500947 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:13:45.500900 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:45.653765 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:13:45.653687 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:51.936504 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:13:51.936469 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:13:52.037353 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:13:52.037319 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:14:07.996150 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:14:07.996069 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:14:15.804358 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:14:15.804325 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:14:26.400052 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:14:26.400013 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:14:34.094232 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:14:34.094194 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:14:45.502024 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:14:45.501961 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:14:55.015406 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:14:55.015372 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:15:06.207906 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:15:06.204361 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:15:15.999675 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:15:15.999642 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:15:33.907882 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:15:33.907851 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:15:33.908397 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:15:33.908120 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:15:38.494688 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:15:38.494647 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:15:47.996345 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:15:47.996305 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:15:57.870098 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:15:57.870058 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:10.701100 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:16:10.701057 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:16:15.216332 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:16:15.216292 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:21.684868 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:16:21.684828 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:16:37.985206 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:16:37.985156 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:16:47.785337 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:16:47.785297 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:16:51.147059 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:16:51.147024 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:16:57.885634 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:16:57.885551 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:17:06.586580 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:17:06.586530 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:17:16.086268 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:17:16.086237 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:19:08.087838 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:19:08.087803 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:19:16.491329 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:19:16.491291 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:19:31.293014 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:19:31.292979 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:19:41.986041 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:19:41.986004 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:19:53.585084 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:19:53.585045 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:20:03.283863 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:20:03.283824 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:20:13.002902 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:20:13.002855 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:20:18.949528 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:20:18.949491 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:20:19.037901 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:20:19.037867 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:20:23.181356 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:20:23.181321 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:20:33.782460 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:20:33.782427 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:20:33.937765 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:20:33.937741 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:20:33.939101 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:20:33.939079 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:20:43.102439 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:20:43.102394 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:20:54.140994 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:20:54.140952 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:21:14.932282 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:21:14.932248 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:21:18.024582 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:21:18.024544 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:21:25.685209 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:21:25.685166 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:21:35.189267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:21:35.189223 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:21:45.087813 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:21:45.087771 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:21:55.880414 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:21:55.880371 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:22:07.484353 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:22:07.484321 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:22:21.398199 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:22:21.398160 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:22:30.987903 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:22:30.987864 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:22:45.383780 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:22:45.383742 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:22:54.582811 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:22:54.582724 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:23:07.084183 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:23:07.084143 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:23:16.488594 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:23:16.488556 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:23:25.986078 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:23:25.986043 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:23:36.580593 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:23:36.580543 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:23:48.484362 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:23:48.484327 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:23:57.587432 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:23:57.587395 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:24:09.888073 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:24:09.888033 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:24:22.395101 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:24:22.395066 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:24:36.285846 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:24:36.285805 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:24:39.448434 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:24:39.448400 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:45.385597 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:24:45.385564 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:24:48.001779 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:24:48.001743 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:48.946757 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:24:48.946725 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:57.190945 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:24:57.190887 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:24:58.290702 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:24:58.290658 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:25:02.285588 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:02.285548 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:06.102120 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:06.102085 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:06.203270 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:06.203231 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:06.522346 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:06.522305 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:06.539911 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:06.539887 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:25:12.386310 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:12.386267 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:12.563095 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:12.563009 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:12.651202 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:12.651088 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:13.765079 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:13.765042 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:13.865124 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:13.865087 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:14.112262 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:14.112174 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:14.317208 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:14.317148 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:15.405725 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:15.405695 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:15.505954 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:15.505901 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:16.032430 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:16.032398 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:19.211896 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:19.211860 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:19.443575 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:19.443542 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:19.543720 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:19.543660 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:19.693660 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:19.693630 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:19.790181 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:19.790143 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:25:20.718481 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:20.718418 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:25.648802 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:25.648764 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:25.766411 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:25.766376 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:25.904149 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:25.904050 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:25.997330 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:25.997292 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:26.097208 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:26.097168 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:26.197519 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:26.197432 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:26.891991 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:26.891956 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:25:27.383165 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:27.383127 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:27.513065 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:27.513023 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:27.945079 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:27.945041 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:28.043706 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:28.043672 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:29.440219 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:29.440183 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:31.842638 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:31.842597 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:32.006617 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:32.006575 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:32.108802 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:32.108715 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:32.230929 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:32.230897 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:33.654910 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:33.654875 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:33.975886 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:33.975805 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:25:33.977730 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:33.977707 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:25:38.125573 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:38.125524 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:38.207388 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:38.207359 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:38.332097 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:38.332067 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:38.434174 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:38.434094 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:39.438882 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:39.438850 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:39.537196 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:39.537164 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:39.641697 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:39.641659 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:39.782495 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:39.782465 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:40.085904 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:40.085821 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:40.185873 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:40.185839 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:41.502607 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:41.502562 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:43.406405 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:43.406364 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:25:44.527492 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:44.527455 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:44.941802 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:44.941716 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:50.543047 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:50.543011 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:50.639790 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:50.639757 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:52.405282 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:52.405251 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:55.182584 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:55.182546 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:25:56.914046 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:56.914013 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:57.015972 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:57.015901 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:57.129168 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:57.129121 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:57.187108 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:57.187012 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:58.220997 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:58.220963 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:58.323635 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:58.323598 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:58.500021 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:58.499986 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:58.689775 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:58.689744 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:59.093096 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:59.093059 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:25:59.193244 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:25:59.193204 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:00.904346 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:00.904297 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:02.665610 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:02.665562 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:02.855708 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:02.855657 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:02.946518 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:02.946415 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:03.049158 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:03.049124 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:04.577610 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:04.577565 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:08.905223 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:08.905168 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:09.005773 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:09.005722 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:09.406551 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:09.406513 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:09.492771 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:09.492733 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:09.582471 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:09.582433 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:10.940680 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:10.940641 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:16.953889 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:16.953842 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:18.305388 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:18.305347 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:30.086116 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:30.086075 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:30.149906 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:30.149879 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:43.589324 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:43.589285 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:49.653128 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:49.653078 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:49.714293 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:49.714255 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:49.791160 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:49.791121 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:26:49.891155 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:26:49.891124 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:05.694436 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:05.694400 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:11.845018 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:11.844985 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:11.956928 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:11.956888 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:12.046612 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:12.046581 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:22.562653 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:22.562608 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:23.370752 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:23.370718 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:29.273624 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:29.273586 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:29.331208 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:29.331170 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:29.405817 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:29.405779 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:29.506555 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:29.506515 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:29.606107 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:29.606006 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:30.294078 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:30.293983 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:30.381114 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:30.381082 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:30.481870 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:30.481833 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:32.093473 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:32.093432 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:32.695655 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:32.695618 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:27:38.097611 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:38.097578 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:38.198331 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:38.198298 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:38.762280 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:38.762207 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:42.200916 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:42.200882 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:27:44.933236 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:44.933201 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:45.037442 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:45.037411 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:45.172647 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:45.172618 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:45.471457 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:45.471418 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:45.766928 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:45.766887 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:48.774330 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:48.774291 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:48.863100 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:48.863060 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:48.929178 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:48.929143 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:49.033504 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:49.033420 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:49.368578 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:49.368487 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:49.479361 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:49.479323 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:53.250454 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:53.250423 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:53.874317 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:53.874284 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:53.979503 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:53.979470 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:27:54.192496 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:54.192412 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:27:54.287202 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:27:54.287157 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:00.203854 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:00.203817 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:00.303599 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:00.303564 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:00.583162 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:00.583129 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:00.979035 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:00.978927 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:01.080507 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:01.080474 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:04.690668 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:04.690641 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:28:06.804803 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:06.804747 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:06.859321 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:06.859290 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:07.087263 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:07.087183 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:07.283787 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:07.283749 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:07.390809 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:07.390719 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:07.491109 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:07.491074 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:07.604100 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:07.604057 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:13.506270 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:13.506234 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:13.610490 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:13.610456 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:13.706466 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:13.706433 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:13.884126 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:13.884031 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:19.813815 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:19.813778 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:19.914256 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:19.914221 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:20.039318 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:20.039288 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:20.137363 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:20.137275 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:25.147162 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:25.147127 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:26.517259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:26.517218 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:26.820053 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:26.819961 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:27.397831 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:27.397802 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:29.998422 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:29.998387 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:30.099567 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:30.099530 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:30.224244 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:30.224209 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:30.367050 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:30.366957 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:30.670998 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:30.670884 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:30.771340 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:30.771306 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:34.470101 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:34.470059 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:36.082698 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:36.082662 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:28:47.288499 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.288465 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6"] Jul 10 19:28:47.291719 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.291700 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.294315 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.294291 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3d2538db\"/\"openshift-service-ca.crt\"" Jul 10 19:28:47.294443 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.294346 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3d2538db\"/\"kube-root-ca.crt\"" Jul 10 19:28:47.295142 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.295126 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3d2538db\"/\"default-dockercfg-ssx4l\"" Jul 10 19:28:47.295228 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.295212 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3d2538db\"/\"e2e-lifecycle-model-3d2538db-kserve-self-signed-certs\"" Jul 10 19:28:47.300599 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.300578 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6"] Jul 10 19:28:47.320259 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.320227 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:28:47.351154 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.351102 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0ae540e-ba86-4e6d-a266-8b9499537c29-tls-certs\") pod \"e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.351285 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.351184 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-tmp-dir\") pod \"e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.351285 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.351216 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-home\") pod \"e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.351285 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.351252 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-model-cache\") pod \"e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.351388 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.351291 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-dshm\") pod \"e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.351388 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.351320 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmngl\" (UniqueName: \"kubernetes.io/projected/b0ae540e-ba86-4e6d-a266-8b9499537c29-kube-api-access-nmngl\") pod \"e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.452648 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.452614 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-home\") pod \"e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.452813 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.452657 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-model-cache\") pod \"e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.452813 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.452689 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-dshm\") pod \"e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.452813 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.452798 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmngl\" (UniqueName: \"kubernetes.io/projected/b0ae540e-ba86-4e6d-a266-8b9499537c29-kube-api-access-nmngl\") pod \"e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.453044 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.452851 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0ae540e-ba86-4e6d-a266-8b9499537c29-tls-certs\") pod \"e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.453044 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.452903 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-tmp-dir\") pod \"e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.453148 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.453121 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-home\") pod \"e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.453209 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.453177 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-model-cache\") pod \"e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.453209 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.453201 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-tmp-dir\") pod \"e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.455248 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.455224 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-dshm\") pod \"e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.455367 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.455352 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0ae540e-ba86-4e6d-a266-8b9499537c29-tls-certs\") pod \"e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.461097 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.461075 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmngl\" (UniqueName: \"kubernetes.io/projected/b0ae540e-ba86-4e6d-a266-8b9499537c29-kube-api-access-nmngl\") pod \"e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.601446 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.601350 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:47.738269 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:47.738236 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6"] Jul 10 19:28:47.740057 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:28:47.740031 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0ae540e_ba86_4e6d_a266_8b9499537c29.slice/crio-da1bccd44a9a145f5005b20811348b8113f18d807daec3cb8bf537cb5799e2a7 WatchSource:0}: Error finding container da1bccd44a9a145f5005b20811348b8113f18d807daec3cb8bf537cb5799e2a7: Status 404 returned error can't find the container with id da1bccd44a9a145f5005b20811348b8113f18d807daec3cb8bf537cb5799e2a7 Jul 10 19:28:48.030907 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:48.030857 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" event={"ID":"b0ae540e-ba86-4e6d-a266-8b9499537c29","Type":"ContainerStarted","Data":"22e88d709bfbdf72c08d5430fc790f942a73bf35f5361519760a3559caddad7b"} Jul 10 19:28:48.030907 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:48.030904 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" event={"ID":"b0ae540e-ba86-4e6d-a266-8b9499537c29","Type":"ContainerStarted","Data":"da1bccd44a9a145f5005b20811348b8113f18d807daec3cb8bf537cb5799e2a7"} Jul 10 19:28:48.050798 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:48.050753 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" podStartSLOduration=1.05073759 podStartE2EDuration="1.05073759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:28:48.048109226 +0000 UTC m=+1694.649251362" watchObservedRunningTime="2026-07-10 19:28:48.05073759 +0000 UTC m=+1694.651879725" Jul 10 19:28:50.980389 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:50.980350 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:28:57.602342 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:57.602302 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:57.602733 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:57.602379 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:57.614993 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:57.614966 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:28:58.096099 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:28:58.096065 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:29:11.884471 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:11.884434 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:29:16.990960 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:16.990901 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:17.381795 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:17.381761 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:21.885743 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:21.885706 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:29:23.039958 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:23.039902 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:28.020118 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:28.020074 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6"] Jul 10 19:29:28.116217 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:28.116150 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:29:32.289334 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:32.289299 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:29:43.040816 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.040752 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" podUID="b0ae540e-ba86-4e6d-a266-8b9499537c29" containerName="main" containerID="cri-o://22e88d709bfbdf72c08d5430fc790f942a73bf35f5361519760a3559caddad7b" gracePeriod=45 Jul 10 19:29:43.290255 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.290232 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:29:43.302171 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.302114 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-home\") pod \"b0ae540e-ba86-4e6d-a266-8b9499537c29\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " Jul 10 19:29:43.302171 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.302169 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0ae540e-ba86-4e6d-a266-8b9499537c29-tls-certs\") pod \"b0ae540e-ba86-4e6d-a266-8b9499537c29\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " Jul 10 19:29:43.302356 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.302200 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-tmp-dir\") pod \"b0ae540e-ba86-4e6d-a266-8b9499537c29\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " Jul 10 19:29:43.302356 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.302257 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmngl\" (UniqueName: \"kubernetes.io/projected/b0ae540e-ba86-4e6d-a266-8b9499537c29-kube-api-access-nmngl\") pod \"b0ae540e-ba86-4e6d-a266-8b9499537c29\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " Jul 10 19:29:43.302356 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.302300 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-model-cache\") pod \"b0ae540e-ba86-4e6d-a266-8b9499537c29\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " Jul 10 19:29:43.302356 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.302321 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-dshm\") pod \"b0ae540e-ba86-4e6d-a266-8b9499537c29\" (UID: \"b0ae540e-ba86-4e6d-a266-8b9499537c29\") " Jul 10 19:29:43.302742 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.302381 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-home" (OuterVolumeSpecName: "home") pod "b0ae540e-ba86-4e6d-a266-8b9499537c29" (UID: "b0ae540e-ba86-4e6d-a266-8b9499537c29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:29:43.302742 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.302480 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0ae540e-ba86-4e6d-a266-8b9499537c29" (UID: "b0ae540e-ba86-4e6d-a266-8b9499537c29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:29:43.302742 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.302659 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-home\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:29:43.302742 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.302677 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0ae540e-ba86-4e6d-a266-8b9499537c29" (UID: "b0ae540e-ba86-4e6d-a266-8b9499537c29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:29:43.302742 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.302700 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-tmp-dir\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:29:43.304722 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.304695 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ae540e-ba86-4e6d-a266-8b9499537c29-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0ae540e-ba86-4e6d-a266-8b9499537c29" (UID: "b0ae540e-ba86-4e6d-a266-8b9499537c29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:29:43.304838 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.304796 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-dshm" (OuterVolumeSpecName: "dshm") pod "b0ae540e-ba86-4e6d-a266-8b9499537c29" (UID: "b0ae540e-ba86-4e6d-a266-8b9499537c29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:29:43.304894 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.304832 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ae540e-ba86-4e6d-a266-8b9499537c29-kube-api-access-nmngl" (OuterVolumeSpecName: "kube-api-access-nmngl") pod "b0ae540e-ba86-4e6d-a266-8b9499537c29" (UID: "b0ae540e-ba86-4e6d-a266-8b9499537c29"). InnerVolumeSpecName "kube-api-access-nmngl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:29:43.307086 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.307052 2561 generic.go:356] "Generic (PLEG): container finished" podID="b0ae540e-ba86-4e6d-a266-8b9499537c29" containerID="22e88d709bfbdf72c08d5430fc790f942a73bf35f5361519760a3559caddad7b" exitCode=0 Jul 10 19:29:43.307190 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.307092 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" Jul 10 19:29:43.307190 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.307131 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" event={"ID":"b0ae540e-ba86-4e6d-a266-8b9499537c29","Type":"ContainerDied","Data":"22e88d709bfbdf72c08d5430fc790f942a73bf35f5361519760a3559caddad7b"} Jul 10 19:29:43.307190 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.307177 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6" event={"ID":"b0ae540e-ba86-4e6d-a266-8b9499537c29","Type":"ContainerDied","Data":"da1bccd44a9a145f5005b20811348b8113f18d807daec3cb8bf537cb5799e2a7"} Jul 10 19:29:43.307318 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.307199 2561 scope.go:117] "RemoveContainer" containerID="22e88d709bfbdf72c08d5430fc790f942a73bf35f5361519760a3559caddad7b" Jul 10 19:29:43.326368 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.326346 2561 scope.go:117] "RemoveContainer" containerID="22e88d709bfbdf72c08d5430fc790f942a73bf35f5361519760a3559caddad7b" Jul 10 19:29:43.326747 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:29:43.326688 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e88d709bfbdf72c08d5430fc790f942a73bf35f5361519760a3559caddad7b\": container with ID starting with 22e88d709bfbdf72c08d5430fc790f942a73bf35f5361519760a3559caddad7b not found: ID does not exist" containerID="22e88d709bfbdf72c08d5430fc790f942a73bf35f5361519760a3559caddad7b" Jul 10 19:29:43.326869 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.326760 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e88d709bfbdf72c08d5430fc790f942a73bf35f5361519760a3559caddad7b"} err="failed to get container status \"22e88d709bfbdf72c08d5430fc790f942a73bf35f5361519760a3559caddad7b\": rpc error: code = NotFound desc = could not find container \"22e88d709bfbdf72c08d5430fc790f942a73bf35f5361519760a3559caddad7b\": container with ID starting with 22e88d709bfbdf72c08d5430fc790f942a73bf35f5361519760a3559caddad7b not found: ID does not exist" Jul 10 19:29:43.338206 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.338184 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6"] Jul 10 19:29:43.343003 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.342974 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3d2538db/e2e-lifecycle-model-3d2538db-kserve-6f7ff45c95-vbgx6"] Jul 10 19:29:43.403279 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.403243 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0ae540e-ba86-4e6d-a266-8b9499537c29-tls-certs\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:29:43.403279 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.403275 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmngl\" (UniqueName: \"kubernetes.io/projected/b0ae540e-ba86-4e6d-a266-8b9499537c29-kube-api-access-nmngl\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:29:43.403279 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.403286 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-model-cache\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:29:43.403491 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.403295 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0ae540e-ba86-4e6d-a266-8b9499537c29-dshm\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:29:43.964534 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:29:43.964499 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0ae540e-ba86-4e6d-a266-8b9499537c29" path="/var/lib/kubelet/pods/b0ae540e-ba86-4e6d-a266-8b9499537c29/volumes" Jul 10 19:30:05.183972 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:05.183902 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:05.459898 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:05.459786 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:08.184219 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:08.184188 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:30:11.517588 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:11.517552 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:11.572527 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:11.572494 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:11.782399 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:11.782242 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:12.014628 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:12.014595 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:13.023901 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:13.023862 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:13.119624 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:13.119557 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:13.219533 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:13.219466 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:14.053910 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:14.053870 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:14.140245 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:14.140211 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:14.240349 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:14.240312 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:15.552305 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:15.552275 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:17.487441 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:17.487408 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:30:21.420649 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:21.420610 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:21.563659 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:21.563626 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:22.085155 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:22.085106 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:28.115785 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:28.115749 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:28.218606 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:28.218571 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:28.316956 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:28.316903 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:29.194973 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:29.194924 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:30:34.005776 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:34.005749 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:30:34.009159 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:34.009134 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:30:38.786066 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:38.786024 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:39.109189 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:39.109105 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:30:48.139255 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:30:48.139219 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:00.122736 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:00.122697 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:06.413021 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:06.412955 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:06.511739 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:06.511687 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:06.800780 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:06.800747 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:07.169702 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:07.169619 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:07.262430 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:07.262396 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:09.934329 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:09.934292 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:10.047431 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:10.047399 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:10.118341 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:10.118307 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:10.218955 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:10.218850 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:10.745528 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:10.745466 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:10.838970 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:10.838910 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:14.435439 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:14.435402 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:14.786714 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:14.786676 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:15.136258 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:15.136186 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:15.274597 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:15.274556 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:18.022990 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:18.022957 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:18.100050 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:18.100016 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:18.196296 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:18.196259 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:18.295196 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:18.295109 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:18.712109 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:18.712021 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:31:19.491988 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:19.491956 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:22.559078 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:22.559043 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:31.649498 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:31.649464 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:31.755839 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:31.755805 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:32.761706 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:32.761673 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x"] Jul 10 19:31:32.762114 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:32.762005 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0ae540e-ba86-4e6d-a266-8b9499537c29" containerName="main" Jul 10 19:31:32.762114 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:32.762017 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0ae540e-ba86-4e6d-a266-8b9499537c29" containerName="main" Jul 10 19:31:32.762114 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:32.762092 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0ae540e-ba86-4e6d-a266-8b9499537c29" containerName="main" Jul 10 19:31:32.765055 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:32.765037 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:32.771705 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:32.771682 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-40ce2b\"/\"test-model-40ce2b-kserve-self-signed-certs\"" Jul 10 19:31:32.773834 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:32.773810 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-40ce2b\"/\"kube-root-ca.crt\"" Jul 10 19:31:32.774066 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:32.774040 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-40ce2b\"/\"default-dockercfg-xbltz\"" Jul 10 19:31:32.774369 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:32.774349 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-40ce2b\"/\"openshift-service-ca.crt\"" Jul 10 19:31:32.784564 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:32.784538 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x"] Jul 10 19:31:32.801594 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:32.801557 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:32.899426 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:32.899385 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-home\") pod \"test-model-40ce2b-kserve-5665bf7ffc-67g2x\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:32.899617 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:32.899443 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-model-cache\") pod \"test-model-40ce2b-kserve-5665bf7ffc-67g2x\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:32.899617 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:32.899528 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-tmp-dir\") pod \"test-model-40ce2b-kserve-5665bf7ffc-67g2x\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:32.899617 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:32.899558 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-tls-certs\") pod \"test-model-40ce2b-kserve-5665bf7ffc-67g2x\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:32.899617 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:32.899588 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s2hj\" (UniqueName: \"kubernetes.io/projected/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-kube-api-access-9s2hj\") pod \"test-model-40ce2b-kserve-5665bf7ffc-67g2x\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:32.899765 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:32.899662 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-dshm\") pod \"test-model-40ce2b-kserve-5665bf7ffc-67g2x\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:33.000756 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:33.000717 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-home\") pod \"test-model-40ce2b-kserve-5665bf7ffc-67g2x\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:33.000985 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:33.000768 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-model-cache\") pod \"test-model-40ce2b-kserve-5665bf7ffc-67g2x\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:33.000985 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:33.000802 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-tmp-dir\") pod \"test-model-40ce2b-kserve-5665bf7ffc-67g2x\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:33.000985 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:33.000820 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-tls-certs\") pod \"test-model-40ce2b-kserve-5665bf7ffc-67g2x\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:33.000985 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:33.000840 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s2hj\" (UniqueName: \"kubernetes.io/projected/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-kube-api-access-9s2hj\") pod \"test-model-40ce2b-kserve-5665bf7ffc-67g2x\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:33.001250 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:33.001230 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-dshm\") pod \"test-model-40ce2b-kserve-5665bf7ffc-67g2x\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:33.001315 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:33.001273 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-model-cache\") pod \"test-model-40ce2b-kserve-5665bf7ffc-67g2x\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:33.001315 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:33.001227 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-home\") pod \"test-model-40ce2b-kserve-5665bf7ffc-67g2x\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:33.001389 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:33.001339 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-tmp-dir\") pod \"test-model-40ce2b-kserve-5665bf7ffc-67g2x\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:33.003355 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:33.003328 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-dshm\") pod \"test-model-40ce2b-kserve-5665bf7ffc-67g2x\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:33.003749 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:33.003726 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-tls-certs\") pod \"test-model-40ce2b-kserve-5665bf7ffc-67g2x\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:33.009511 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:33.009488 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s2hj\" (UniqueName: \"kubernetes.io/projected/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-kube-api-access-9s2hj\") pod \"test-model-40ce2b-kserve-5665bf7ffc-67g2x\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:33.080159 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:33.080069 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:33.221574 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:33.221546 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x"] Jul 10 19:31:33.223606 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:31:33.223579 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4134ad9a_64e8_4a78_9b24_1c4417bd8e0c.slice/crio-4742d077f78e7ef5b6c73a9aef1ca4a55dfe837abb326dec9ab0eb773cb53ef5 WatchSource:0}: Error finding container 4742d077f78e7ef5b6c73a9aef1ca4a55dfe837abb326dec9ab0eb773cb53ef5: Status 404 returned error can't find the container with id 4742d077f78e7ef5b6c73a9aef1ca4a55dfe837abb326dec9ab0eb773cb53ef5 Jul 10 19:31:33.690389 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:33.690347 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:31:33.840876 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:33.840842 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" event={"ID":"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c","Type":"ContainerStarted","Data":"1962ee39c6b55476005ce3ee1c48362c558f783a3b113cb9ffdb77d909a9a7bd"} Jul 10 19:31:33.840876 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:33.840884 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" event={"ID":"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c","Type":"ContainerStarted","Data":"4742d077f78e7ef5b6c73a9aef1ca4a55dfe837abb326dec9ab0eb773cb53ef5"} Jul 10 19:31:33.858415 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:33.858362 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" podStartSLOduration=1.8583461300000002 podStartE2EDuration="1.85834613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:31:33.856853359 +0000 UTC m=+1860.457995493" watchObservedRunningTime="2026-07-10 19:31:33.85834613 +0000 UTC m=+1860.459488265" Jul 10 19:31:40.065698 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:40.065656 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:40.314015 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:40.313974 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:43.080515 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:43.080474 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:43.080971 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:43.080535 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:43.092978 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:43.092926 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:43.901791 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:43.901756 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:31:46.392370 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:46.392333 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:31:55.395494 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.395415 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b"] Jul 10 19:31:55.398795 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.398779 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.401440 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.401412 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-69b0c1\"/\"openshift-service-ca.crt\"" Jul 10 19:31:55.401564 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.401491 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-69b0c1\"/\"kube-root-ca.crt\"" Jul 10 19:31:55.402552 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.402534 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-69b0c1\"/\"test-model-69b0c1-kserve-self-signed-certs\"" Jul 10 19:31:55.402647 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.402570 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-69b0c1\"/\"default-dockercfg-fkd4d\"" Jul 10 19:31:55.410206 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.410185 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b"] Jul 10 19:31:55.419395 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.419177 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:31:55.502994 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.502958 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-model-cache\") pod \"test-model-69b0c1-kserve-5d869c46c5-2sc4b\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.502994 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.502999 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/778733fa-7a96-4bf3-84fc-42ad056449d5-tls-certs\") pod \"test-model-69b0c1-kserve-5d869c46c5-2sc4b\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.503252 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.503082 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-tmp-dir\") pod \"test-model-69b0c1-kserve-5d869c46c5-2sc4b\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.503252 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.503172 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-home\") pod \"test-model-69b0c1-kserve-5d869c46c5-2sc4b\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.503252 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.503201 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brgsb\" (UniqueName: \"kubernetes.io/projected/778733fa-7a96-4bf3-84fc-42ad056449d5-kube-api-access-brgsb\") pod \"test-model-69b0c1-kserve-5d869c46c5-2sc4b\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.503252 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.503231 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-dshm\") pod \"test-model-69b0c1-kserve-5d869c46c5-2sc4b\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.604386 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.604328 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-model-cache\") pod \"test-model-69b0c1-kserve-5d869c46c5-2sc4b\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.604386 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.604384 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/778733fa-7a96-4bf3-84fc-42ad056449d5-tls-certs\") pod \"test-model-69b0c1-kserve-5d869c46c5-2sc4b\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.604625 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.604411 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-tmp-dir\") pod \"test-model-69b0c1-kserve-5d869c46c5-2sc4b\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.604625 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.604443 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-home\") pod \"test-model-69b0c1-kserve-5d869c46c5-2sc4b\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.604625 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.604462 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brgsb\" (UniqueName: \"kubernetes.io/projected/778733fa-7a96-4bf3-84fc-42ad056449d5-kube-api-access-brgsb\") pod \"test-model-69b0c1-kserve-5d869c46c5-2sc4b\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.604625 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.604487 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-dshm\") pod \"test-model-69b0c1-kserve-5d869c46c5-2sc4b\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.604831 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.604812 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-tmp-dir\") pod \"test-model-69b0c1-kserve-5d869c46c5-2sc4b\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.604894 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.604818 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-model-cache\") pod \"test-model-69b0c1-kserve-5d869c46c5-2sc4b\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.604894 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.604881 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-home\") pod \"test-model-69b0c1-kserve-5d869c46c5-2sc4b\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.606831 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.606810 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-dshm\") pod \"test-model-69b0c1-kserve-5d869c46c5-2sc4b\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.607075 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.607057 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/778733fa-7a96-4bf3-84fc-42ad056449d5-tls-certs\") pod \"test-model-69b0c1-kserve-5d869c46c5-2sc4b\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.614358 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.614333 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brgsb\" (UniqueName: \"kubernetes.io/projected/778733fa-7a96-4bf3-84fc-42ad056449d5-kube-api-access-brgsb\") pod \"test-model-69b0c1-kserve-5d869c46c5-2sc4b\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.708919 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.708809 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:31:55.843802 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.843773 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b"] Jul 10 19:31:55.845587 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:31:55.845559 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod778733fa_7a96_4bf3_84fc_42ad056449d5.slice/crio-4c150f674b980c460dccb9538ad2c303154abeb1c10a8904184e5489427d33fd WatchSource:0}: Error finding container 4c150f674b980c460dccb9538ad2c303154abeb1c10a8904184e5489427d33fd: Status 404 returned error can't find the container with id 4c150f674b980c460dccb9538ad2c303154abeb1c10a8904184e5489427d33fd Jul 10 19:31:55.953081 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.953041 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" event={"ID":"778733fa-7a96-4bf3-84fc-42ad056449d5","Type":"ContainerStarted","Data":"a78c6494fe1e383565ebc2e1a99704ac3b729b7d91ae49fa1c324b05eb8e035c"} Jul 10 19:31:55.953287 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.953092 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" event={"ID":"778733fa-7a96-4bf3-84fc-42ad056449d5","Type":"ContainerStarted","Data":"4c150f674b980c460dccb9538ad2c303154abeb1c10a8904184e5489427d33fd"} Jul 10 19:31:55.992023 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:31:55.991887 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" podStartSLOduration=0.991868215 podStartE2EDuration="991.868215ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:31:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:31:55.988048975 +0000 UTC m=+1882.589191110" watchObservedRunningTime="2026-07-10 19:31:55.991868215 +0000 UTC m=+1882.593010350" Jul 10 19:32:01.299470 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:01.299424 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:32:05.709980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:05.709920 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:32:05.710465 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:05.709993 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:32:05.722716 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:05.722685 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:32:06.017872 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:06.017840 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:32:15.493414 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:15.493377 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:32:26.046980 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:26.046928 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x"] Jul 10 19:32:26.068968 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:26.068896 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:32:26.905064 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:26.905031 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b"] Jul 10 19:32:26.925329 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:26.925291 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:32:27.101432 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:27.101397 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:32:27.501112 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:27.501065 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:32:27.598664 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:27.598625 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:32:27.698902 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:27.698860 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:32:27.821635 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:27.821550 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:32:31.588759 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:31.588725 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:32:32.669863 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:32.669831 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:32:41.071766 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.071704 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" podUID="4134ad9a-64e8-4a78-9b24-1c4417bd8e0c" containerName="main" containerID="cri-o://1962ee39c6b55476005ce3ee1c48362c558f783a3b113cb9ffdb77d909a9a7bd" gracePeriod=45 Jul 10 19:32:41.320408 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.320386 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:32:41.377180 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.377080 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-home\") pod \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " Jul 10 19:32:41.377180 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.377126 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-tls-certs\") pod \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " Jul 10 19:32:41.377180 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.377181 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s2hj\" (UniqueName: \"kubernetes.io/projected/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-kube-api-access-9s2hj\") pod \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " Jul 10 19:32:41.377475 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.377213 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-model-cache\") pod \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " Jul 10 19:32:41.377475 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.377230 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-tmp-dir\") pod \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " Jul 10 19:32:41.377475 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.377330 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-dshm\") pod \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\" (UID: \"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c\") " Jul 10 19:32:41.377475 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.377400 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-home" (OuterVolumeSpecName: "home") pod "4134ad9a-64e8-4a78-9b24-1c4417bd8e0c" (UID: "4134ad9a-64e8-4a78-9b24-1c4417bd8e0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:32:41.377688 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.377511 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4134ad9a-64e8-4a78-9b24-1c4417bd8e0c" (UID: "4134ad9a-64e8-4a78-9b24-1c4417bd8e0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:32:41.377688 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.377528 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4134ad9a-64e8-4a78-9b24-1c4417bd8e0c" (UID: "4134ad9a-64e8-4a78-9b24-1c4417bd8e0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:32:41.377688 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.377622 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-home\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:32:41.377688 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.377638 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-model-cache\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:32:41.377688 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.377651 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-tmp-dir\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:32:41.379735 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.379703 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-kube-api-access-9s2hj" (OuterVolumeSpecName: "kube-api-access-9s2hj") pod "4134ad9a-64e8-4a78-9b24-1c4417bd8e0c" (UID: "4134ad9a-64e8-4a78-9b24-1c4417bd8e0c"). InnerVolumeSpecName "kube-api-access-9s2hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:32:41.379735 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.379707 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4134ad9a-64e8-4a78-9b24-1c4417bd8e0c" (UID: "4134ad9a-64e8-4a78-9b24-1c4417bd8e0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:32:41.379735 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.379716 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-dshm" (OuterVolumeSpecName: "dshm") pod "4134ad9a-64e8-4a78-9b24-1c4417bd8e0c" (UID: "4134ad9a-64e8-4a78-9b24-1c4417bd8e0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:32:41.479080 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.479026 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-dshm\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:32:41.479080 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.479074 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-tls-certs\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:32:41.479080 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.479088 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s2hj\" (UniqueName: \"kubernetes.io/projected/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c-kube-api-access-9s2hj\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:32:41.926129 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:41.926061 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" podUID="778733fa-7a96-4bf3-84fc-42ad056449d5" containerName="main" containerID="cri-o://a78c6494fe1e383565ebc2e1a99704ac3b729b7d91ae49fa1c324b05eb8e035c" gracePeriod=45 Jul 10 19:32:42.176823 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.176770 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:32:42.186535 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.186505 2561 generic.go:356] "Generic (PLEG): container finished" podID="778733fa-7a96-4bf3-84fc-42ad056449d5" containerID="a78c6494fe1e383565ebc2e1a99704ac3b729b7d91ae49fa1c324b05eb8e035c" exitCode=0 Jul 10 19:32:42.186645 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.186540 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" Jul 10 19:32:42.186645 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.186554 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" event={"ID":"778733fa-7a96-4bf3-84fc-42ad056449d5","Type":"ContainerDied","Data":"a78c6494fe1e383565ebc2e1a99704ac3b729b7d91ae49fa1c324b05eb8e035c"} Jul 10 19:32:42.186645 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.186594 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b" event={"ID":"778733fa-7a96-4bf3-84fc-42ad056449d5","Type":"ContainerDied","Data":"4c150f674b980c460dccb9538ad2c303154abeb1c10a8904184e5489427d33fd"} Jul 10 19:32:42.186645 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.186611 2561 scope.go:117] "RemoveContainer" containerID="a78c6494fe1e383565ebc2e1a99704ac3b729b7d91ae49fa1c324b05eb8e035c" Jul 10 19:32:42.187880 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.187856 2561 generic.go:356] "Generic (PLEG): container finished" podID="4134ad9a-64e8-4a78-9b24-1c4417bd8e0c" containerID="1962ee39c6b55476005ce3ee1c48362c558f783a3b113cb9ffdb77d909a9a7bd" exitCode=0 Jul 10 19:32:42.188036 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.187886 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" event={"ID":"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c","Type":"ContainerDied","Data":"1962ee39c6b55476005ce3ee1c48362c558f783a3b113cb9ffdb77d909a9a7bd"} Jul 10 19:32:42.188036 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.187904 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" Jul 10 19:32:42.188036 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.187907 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x" event={"ID":"4134ad9a-64e8-4a78-9b24-1c4417bd8e0c","Type":"ContainerDied","Data":"4742d077f78e7ef5b6c73a9aef1ca4a55dfe837abb326dec9ab0eb773cb53ef5"} Jul 10 19:32:42.195332 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.195316 2561 scope.go:117] "RemoveContainer" containerID="a78c6494fe1e383565ebc2e1a99704ac3b729b7d91ae49fa1c324b05eb8e035c" Jul 10 19:32:42.195599 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:32:42.195579 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a78c6494fe1e383565ebc2e1a99704ac3b729b7d91ae49fa1c324b05eb8e035c\": container with ID starting with a78c6494fe1e383565ebc2e1a99704ac3b729b7d91ae49fa1c324b05eb8e035c not found: ID does not exist" containerID="a78c6494fe1e383565ebc2e1a99704ac3b729b7d91ae49fa1c324b05eb8e035c" Jul 10 19:32:42.195707 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.195609 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78c6494fe1e383565ebc2e1a99704ac3b729b7d91ae49fa1c324b05eb8e035c"} err="failed to get container status \"a78c6494fe1e383565ebc2e1a99704ac3b729b7d91ae49fa1c324b05eb8e035c\": rpc error: code = NotFound desc = could not find container \"a78c6494fe1e383565ebc2e1a99704ac3b729b7d91ae49fa1c324b05eb8e035c\": container with ID starting with a78c6494fe1e383565ebc2e1a99704ac3b729b7d91ae49fa1c324b05eb8e035c not found: ID does not exist" Jul 10 19:32:42.195707 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.195635 2561 scope.go:117] "RemoveContainer" containerID="1962ee39c6b55476005ce3ee1c48362c558f783a3b113cb9ffdb77d909a9a7bd" Jul 10 19:32:42.204164 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.204147 2561 scope.go:117] "RemoveContainer" containerID="1962ee39c6b55476005ce3ee1c48362c558f783a3b113cb9ffdb77d909a9a7bd" Jul 10 19:32:42.204459 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:32:42.204441 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1962ee39c6b55476005ce3ee1c48362c558f783a3b113cb9ffdb77d909a9a7bd\": container with ID starting with 1962ee39c6b55476005ce3ee1c48362c558f783a3b113cb9ffdb77d909a9a7bd not found: ID does not exist" containerID="1962ee39c6b55476005ce3ee1c48362c558f783a3b113cb9ffdb77d909a9a7bd" Jul 10 19:32:42.204519 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.204466 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1962ee39c6b55476005ce3ee1c48362c558f783a3b113cb9ffdb77d909a9a7bd"} err="failed to get container status \"1962ee39c6b55476005ce3ee1c48362c558f783a3b113cb9ffdb77d909a9a7bd\": rpc error: code = NotFound desc = could not find container \"1962ee39c6b55476005ce3ee1c48362c558f783a3b113cb9ffdb77d909a9a7bd\": container with ID starting with 1962ee39c6b55476005ce3ee1c48362c558f783a3b113cb9ffdb77d909a9a7bd not found: ID does not exist" Jul 10 19:32:42.208989 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.208964 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x"] Jul 10 19:32:42.213573 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.213545 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-40ce2b/test-model-40ce2b-kserve-5665bf7ffc-67g2x"] Jul 10 19:32:42.284854 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.284807 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-model-cache\") pod \"778733fa-7a96-4bf3-84fc-42ad056449d5\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " Jul 10 19:32:42.284854 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.284855 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-dshm\") pod \"778733fa-7a96-4bf3-84fc-42ad056449d5\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " Jul 10 19:32:42.285133 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.284888 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brgsb\" (UniqueName: \"kubernetes.io/projected/778733fa-7a96-4bf3-84fc-42ad056449d5-kube-api-access-brgsb\") pod \"778733fa-7a96-4bf3-84fc-42ad056449d5\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " Jul 10 19:32:42.285133 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.284916 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/778733fa-7a96-4bf3-84fc-42ad056449d5-tls-certs\") pod \"778733fa-7a96-4bf3-84fc-42ad056449d5\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " Jul 10 19:32:42.285133 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.285050 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-tmp-dir\") pod \"778733fa-7a96-4bf3-84fc-42ad056449d5\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " Jul 10 19:32:42.285133 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.285082 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-home\") pod \"778733fa-7a96-4bf3-84fc-42ad056449d5\" (UID: \"778733fa-7a96-4bf3-84fc-42ad056449d5\") " Jul 10 19:32:42.285348 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.285193 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "778733fa-7a96-4bf3-84fc-42ad056449d5" (UID: "778733fa-7a96-4bf3-84fc-42ad056449d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:32:42.285348 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.285318 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "778733fa-7a96-4bf3-84fc-42ad056449d5" (UID: "778733fa-7a96-4bf3-84fc-42ad056449d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:32:42.285448 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.285378 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-home" (OuterVolumeSpecName: "home") pod "778733fa-7a96-4bf3-84fc-42ad056449d5" (UID: "778733fa-7a96-4bf3-84fc-42ad056449d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:32:42.285448 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.285401 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-tmp-dir\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:32:42.285448 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.285417 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-model-cache\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:32:42.287323 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.287293 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778733fa-7a96-4bf3-84fc-42ad056449d5-kube-api-access-brgsb" (OuterVolumeSpecName: "kube-api-access-brgsb") pod "778733fa-7a96-4bf3-84fc-42ad056449d5" (UID: "778733fa-7a96-4bf3-84fc-42ad056449d5"). InnerVolumeSpecName "kube-api-access-brgsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:32:42.287323 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.287302 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-dshm" (OuterVolumeSpecName: "dshm") pod "778733fa-7a96-4bf3-84fc-42ad056449d5" (UID: "778733fa-7a96-4bf3-84fc-42ad056449d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:32:42.287501 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.287358 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778733fa-7a96-4bf3-84fc-42ad056449d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "778733fa-7a96-4bf3-84fc-42ad056449d5" (UID: "778733fa-7a96-4bf3-84fc-42ad056449d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:32:42.386160 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.386105 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-dshm\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:32:42.386160 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.386155 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brgsb\" (UniqueName: \"kubernetes.io/projected/778733fa-7a96-4bf3-84fc-42ad056449d5-kube-api-access-brgsb\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:32:42.386160 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.386165 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/778733fa-7a96-4bf3-84fc-42ad056449d5-tls-certs\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:32:42.386160 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.386173 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/778733fa-7a96-4bf3-84fc-42ad056449d5-home\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:32:42.508088 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.508057 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b"] Jul 10 19:32:42.511467 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:42.511444 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-69b0c1/test-model-69b0c1-kserve-5d869c46c5-2sc4b"] Jul 10 19:32:43.964028 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:43.963985 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4134ad9a-64e8-4a78-9b24-1c4417bd8e0c" path="/var/lib/kubelet/pods/4134ad9a-64e8-4a78-9b24-1c4417bd8e0c/volumes" Jul 10 19:32:43.964551 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:32:43.964529 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="778733fa-7a96-4bf3-84fc-42ad056449d5" path="/var/lib/kubelet/pods/778733fa-7a96-4bf3-84fc-42ad056449d5/volumes" Jul 10 19:33:09.816357 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:09.816318 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:10.056387 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:10.056353 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:10.274037 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:10.274005 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:10.429208 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:10.429172 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:10.800738 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:10.800701 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:10.994628 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:10.994591 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:11.095301 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:11.095204 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:11.257304 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:11.257269 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:14.386825 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:14.386793 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:33:16.796104 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:16.796074 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:16.898430 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:16.898397 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:17.054276 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:17.054179 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:17.264724 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:17.264691 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:17.577429 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:17.577389 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:17.864306 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:17.864232 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:18.017151 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:18.017120 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:20.440598 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:20.440570 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:20.539521 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:20.539490 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:20.904239 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:20.904207 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:21.061877 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:21.061822 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:21.422758 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:21.422725 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:22.020411 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:22.020378 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:22.122451 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:22.122422 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:22.222981 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:22.222921 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:24.890119 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:24.890029 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:25.327050 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:25.327017 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:25.619754 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:25.619661 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:26.486166 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:26.486133 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:28.188987 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:28.188948 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:33:28.515595 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:28.515559 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:28.797330 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:28.797250 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:28.905078 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:28.905049 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:29.079165 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:29.079081 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:29.193345 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:29.193313 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:32.994170 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:32.994134 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:40.943262 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:40.943223 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:41.281729 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:41.281691 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:41.554878 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:41.554786 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:41.658123 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:41.658083 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:41.907628 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:41.907541 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:46.214173 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:46.214140 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:33:47.801715 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:47.801669 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:47.874115 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:47.874067 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:48.051350 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:48.051317 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:48.311211 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:48.311177 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:48.490451 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:48.490412 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:48.824625 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:48.824586 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:48.974388 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:48.974357 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:49.074576 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:49.074545 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:49.235735 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:49.235650 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:54.671602 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:54.671571 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:54.772316 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:54.772277 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:54.935026 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:54.933858 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:55.156179 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:55.156139 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:55.487808 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:55.487770 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:55.769547 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:55.769509 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:56.351105 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:56.351069 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:56.964615 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:56.964572 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:57.023184 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:57.023151 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:58.741531 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:58.741496 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:58.827139 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:58.827108 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:58.897436 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:58.897399 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:59.145130 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:59.145092 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:59.393862 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:59.393831 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:33:59.508847 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:33:59.508815 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:00.185771 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:00.185734 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:34:03.216018 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:03.215982 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:03.540569 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:03.540539 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:03.818924 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:03.818826 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:04.878844 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:04.878803 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:06.747052 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:06.747017 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:06.847666 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:06.847629 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:06.908282 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:06.908250 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:07.006005 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:07.005898 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:07.461563 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:07.461478 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:07.559075 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:07.559045 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:11.242109 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:11.242069 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:11.384103 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:11.384059 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:34:20.146626 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:20.146591 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:20.244658 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:20.244623 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:28.422552 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:28.422509 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:28.523182 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:28.523142 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:29.080440 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.080402 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb"] Jul 10 19:34:29.080819 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.080799 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4134ad9a-64e8-4a78-9b24-1c4417bd8e0c" containerName="main" Jul 10 19:34:29.080905 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.080821 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4134ad9a-64e8-4a78-9b24-1c4417bd8e0c" containerName="main" Jul 10 19:34:29.080905 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.080834 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="778733fa-7a96-4bf3-84fc-42ad056449d5" containerName="main" Jul 10 19:34:29.080905 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.080842 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="778733fa-7a96-4bf3-84fc-42ad056449d5" containerName="main" Jul 10 19:34:29.081086 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.080954 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="778733fa-7a96-4bf3-84fc-42ad056449d5" containerName="main" Jul 10 19:34:29.081086 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.080984 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4134ad9a-64e8-4a78-9b24-1c4417bd8e0c" containerName="main" Jul 10 19:34:29.084102 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.084076 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.086593 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.086567 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b1f450\"/\"kube-root-ca.crt\"" Jul 10 19:34:29.087537 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.087510 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b1f450\"/\"auth-test-model-b1f450-kserve-self-signed-certs\"" Jul 10 19:34:29.087537 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.087523 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b1f450\"/\"openshift-service-ca.crt\"" Jul 10 19:34:29.087707 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.087589 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b1f450\"/\"default-dockercfg-6gflp\"" Jul 10 19:34:29.093480 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.093454 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb"] Jul 10 19:34:29.172670 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.172630 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:29.256623 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.256588 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-tls-certs\") pod \"auth-test-model-b1f450-kserve-7cc9d454dc-bphqb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.256820 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.256630 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-model-cache\") pod \"auth-test-model-b1f450-kserve-7cc9d454dc-bphqb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.256820 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.256675 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-home\") pod \"auth-test-model-b1f450-kserve-7cc9d454dc-bphqb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.256820 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.256692 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62lnc\" (UniqueName: \"kubernetes.io/projected/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-kube-api-access-62lnc\") pod \"auth-test-model-b1f450-kserve-7cc9d454dc-bphqb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.256820 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.256709 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-dshm\") pod \"auth-test-model-b1f450-kserve-7cc9d454dc-bphqb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.256820 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.256728 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-tmp-dir\") pod \"auth-test-model-b1f450-kserve-7cc9d454dc-bphqb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.357897 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.357798 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-home\") pod \"auth-test-model-b1f450-kserve-7cc9d454dc-bphqb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.357897 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.357837 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62lnc\" (UniqueName: \"kubernetes.io/projected/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-kube-api-access-62lnc\") pod \"auth-test-model-b1f450-kserve-7cc9d454dc-bphqb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.357897 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.357857 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-dshm\") pod \"auth-test-model-b1f450-kserve-7cc9d454dc-bphqb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.358226 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.357992 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-tmp-dir\") pod \"auth-test-model-b1f450-kserve-7cc9d454dc-bphqb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.358226 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.358082 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-tls-certs\") pod \"auth-test-model-b1f450-kserve-7cc9d454dc-bphqb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.358226 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.358112 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-model-cache\") pod \"auth-test-model-b1f450-kserve-7cc9d454dc-bphqb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.358367 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.358295 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-home\") pod \"auth-test-model-b1f450-kserve-7cc9d454dc-bphqb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.358367 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.358348 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-tmp-dir\") pod \"auth-test-model-b1f450-kserve-7cc9d454dc-bphqb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.358468 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.358411 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-model-cache\") pod \"auth-test-model-b1f450-kserve-7cc9d454dc-bphqb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.360461 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.360434 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-dshm\") pod \"auth-test-model-b1f450-kserve-7cc9d454dc-bphqb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.360748 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.360728 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-tls-certs\") pod \"auth-test-model-b1f450-kserve-7cc9d454dc-bphqb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.365996 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.365975 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62lnc\" (UniqueName: \"kubernetes.io/projected/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-kube-api-access-62lnc\") pod \"auth-test-model-b1f450-kserve-7cc9d454dc-bphqb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.394504 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.394457 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:29.543014 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.542985 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb"] Jul 10 19:34:29.545807 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:34:29.545779 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda27b7205_eaa0_4156_9c75_94d31a6d9eeb.slice/crio-a7bec1980b54ff6b1a2d7b8492320bb9bd2653a860f438f5e0cde1c9c31a9adf WatchSource:0}: Error finding container a7bec1980b54ff6b1a2d7b8492320bb9bd2653a860f438f5e0cde1c9c31a9adf: Status 404 returned error can't find the container with id a7bec1980b54ff6b1a2d7b8492320bb9bd2653a860f438f5e0cde1c9c31a9adf Jul 10 19:34:29.711073 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.710973 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" event={"ID":"a27b7205-eaa0-4156-9c75-94d31a6d9eeb","Type":"ContainerStarted","Data":"9de89fd1b345db1bebdb6a8e758b9a15dd99d75d29f25a1ff0014206a4c8246d"} Jul 10 19:34:29.711073 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.711026 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" event={"ID":"a27b7205-eaa0-4156-9c75-94d31a6d9eeb","Type":"ContainerStarted","Data":"a7bec1980b54ff6b1a2d7b8492320bb9bd2653a860f438f5e0cde1c9c31a9adf"} Jul 10 19:34:29.728357 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:29.728306 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" podStartSLOduration=0.728291317 podStartE2EDuration="728.291317ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:34:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:34:29.727644418 +0000 UTC m=+2036.328786554" watchObservedRunningTime="2026-07-10 19:34:29.728291317 +0000 UTC m=+2036.329433451" Jul 10 19:34:32.583892 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:32.583852 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:34:39.395383 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:39.395324 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:39.395383 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:39.395394 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:39.407695 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:39.407668 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:39.766831 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:39.766799 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:34:54.887988 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:54.887887 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h"] Jul 10 19:34:54.891005 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:54.890984 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:54.893376 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:54.893342 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b02417\"/\"openshift-service-ca.crt\"" Jul 10 19:34:54.894541 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:54.894518 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b02417\"/\"default-dockercfg-nflsl\"" Jul 10 19:34:54.894541 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:54.894537 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b02417\"/\"kube-root-ca.crt\"" Jul 10 19:34:54.894736 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:54.894520 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b02417\"/\"auth-test-model-b02417-kserve-self-signed-certs\"" Jul 10 19:34:54.898147 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:54.898122 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h"] Jul 10 19:34:54.916149 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:54.916124 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:34:54.968364 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:54.968330 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/252e3e6a-3ca9-413f-a884-ea0d618511ff-tls-certs\") pod \"auth-test-model-b02417-kserve-7f64b76655-pn97h\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:54.968549 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:54.968386 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-model-cache\") pod \"auth-test-model-b02417-kserve-7f64b76655-pn97h\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:54.968549 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:54.968422 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlnpc\" (UniqueName: \"kubernetes.io/projected/252e3e6a-3ca9-413f-a884-ea0d618511ff-kube-api-access-nlnpc\") pod \"auth-test-model-b02417-kserve-7f64b76655-pn97h\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:54.968549 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:54.968452 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-home\") pod \"auth-test-model-b02417-kserve-7f64b76655-pn97h\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:54.968549 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:54.968500 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-dshm\") pod \"auth-test-model-b02417-kserve-7f64b76655-pn97h\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:54.968713 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:54.968541 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-tmp-dir\") pod \"auth-test-model-b02417-kserve-7f64b76655-pn97h\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:55.069186 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:55.069144 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-tmp-dir\") pod \"auth-test-model-b02417-kserve-7f64b76655-pn97h\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:55.069186 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:55.069192 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/252e3e6a-3ca9-413f-a884-ea0d618511ff-tls-certs\") pod \"auth-test-model-b02417-kserve-7f64b76655-pn97h\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:55.069451 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:55.069217 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-model-cache\") pod \"auth-test-model-b02417-kserve-7f64b76655-pn97h\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:55.069451 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:55.069334 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlnpc\" (UniqueName: \"kubernetes.io/projected/252e3e6a-3ca9-413f-a884-ea0d618511ff-kube-api-access-nlnpc\") pod \"auth-test-model-b02417-kserve-7f64b76655-pn97h\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:55.069451 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:55.069372 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-home\") pod \"auth-test-model-b02417-kserve-7f64b76655-pn97h\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:55.069451 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:55.069404 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-dshm\") pod \"auth-test-model-b02417-kserve-7f64b76655-pn97h\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:55.069663 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:55.069573 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-tmp-dir\") pod \"auth-test-model-b02417-kserve-7f64b76655-pn97h\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:55.069663 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:55.069619 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-model-cache\") pod \"auth-test-model-b02417-kserve-7f64b76655-pn97h\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:55.069786 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:55.069663 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-home\") pod \"auth-test-model-b02417-kserve-7f64b76655-pn97h\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:55.071657 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:55.071631 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-dshm\") pod \"auth-test-model-b02417-kserve-7f64b76655-pn97h\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:55.071865 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:55.071844 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/252e3e6a-3ca9-413f-a884-ea0d618511ff-tls-certs\") pod \"auth-test-model-b02417-kserve-7f64b76655-pn97h\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:55.077281 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:55.077261 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlnpc\" (UniqueName: \"kubernetes.io/projected/252e3e6a-3ca9-413f-a884-ea0d618511ff-kube-api-access-nlnpc\") pod \"auth-test-model-b02417-kserve-7f64b76655-pn97h\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:55.201756 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:55.201646 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:34:55.337866 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:55.337837 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h"] Jul 10 19:34:55.339947 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:34:55.339895 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod252e3e6a_3ca9_413f_a884_ea0d618511ff.slice/crio-315303d4ea86d68bcd000737299cee30446cde7fe2c7e47af7f1890d25dbe023 WatchSource:0}: Error finding container 315303d4ea86d68bcd000737299cee30446cde7fe2c7e47af7f1890d25dbe023: Status 404 returned error can't find the container with id 315303d4ea86d68bcd000737299cee30446cde7fe2c7e47af7f1890d25dbe023 Jul 10 19:34:55.838771 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:55.838730 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" event={"ID":"252e3e6a-3ca9-413f-a884-ea0d618511ff","Type":"ContainerStarted","Data":"97066b77fca0a75ad75d42a071575aa8acaed7a1216570a0d6aea3ed001b1cdd"} Jul 10 19:34:55.838973 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:55.838783 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" event={"ID":"252e3e6a-3ca9-413f-a884-ea0d618511ff","Type":"ContainerStarted","Data":"315303d4ea86d68bcd000737299cee30446cde7fe2c7e47af7f1890d25dbe023"} Jul 10 19:34:55.860846 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:55.860790 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" podStartSLOduration=1.8607754349999999 podStartE2EDuration="1.860775435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:34:55.857210435 +0000 UTC m=+2062.458352570" watchObservedRunningTime="2026-07-10 19:34:55.860775435 +0000 UTC m=+2062.461917607" Jul 10 19:34:58.694894 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:34:58.694850 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:35:05.201917 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:35:05.201868 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:35:05.202367 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:35:05.201978 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:35:05.214751 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:35:05.214722 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:35:05.902365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:35:05.902333 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:35:13.392915 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:35:13.392875 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:35:14.379212 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:35:14.379177 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:35:31.282437 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:35:31.282399 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:35:34.038317 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:35:34.038291 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:35:34.043068 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:35:34.043039 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:35:48.890201 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:35:48.890159 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:36:08.480757 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:36:08.480716 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:36:23.887452 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:36:23.887353 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:36:43.389558 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:36:43.389524 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:36:57.284072 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:36:57.284038 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb"] Jul 10 19:36:57.340635 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:36:57.340594 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:36:58.265013 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:36:58.264975 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h"] Jul 10 19:36:58.446131 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:36:58.446089 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:00.990640 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:00.990594 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:37:12.304476 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.304416 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" podUID="a27b7205-eaa0-4156-9c75-94d31a6d9eeb" containerName="main" containerID="cri-o://9de89fd1b345db1bebdb6a8e758b9a15dd99d75d29f25a1ff0014206a4c8246d" gracePeriod=45 Jul 10 19:37:12.516341 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.516309 2561 generic.go:356] "Generic (PLEG): container finished" podID="a27b7205-eaa0-4156-9c75-94d31a6d9eeb" containerID="9de89fd1b345db1bebdb6a8e758b9a15dd99d75d29f25a1ff0014206a4c8246d" exitCode=0 Jul 10 19:37:12.516508 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.516378 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" event={"ID":"a27b7205-eaa0-4156-9c75-94d31a6d9eeb","Type":"ContainerDied","Data":"9de89fd1b345db1bebdb6a8e758b9a15dd99d75d29f25a1ff0014206a4c8246d"} Jul 10 19:37:12.551777 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.551754 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:37:12.643013 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.642872 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-home\") pod \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " Jul 10 19:37:12.643013 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.642965 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62lnc\" (UniqueName: \"kubernetes.io/projected/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-kube-api-access-62lnc\") pod \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " Jul 10 19:37:12.643013 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.643010 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-tls-certs\") pod \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " Jul 10 19:37:12.643293 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.643057 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-model-cache\") pod \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " Jul 10 19:37:12.643293 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.643080 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-tmp-dir\") pod \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " Jul 10 19:37:12.643293 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.643119 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-dshm\") pod \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\" (UID: \"a27b7205-eaa0-4156-9c75-94d31a6d9eeb\") " Jul 10 19:37:12.643293 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.643224 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-home" (OuterVolumeSpecName: "home") pod "a27b7205-eaa0-4156-9c75-94d31a6d9eeb" (UID: "a27b7205-eaa0-4156-9c75-94d31a6d9eeb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:37:12.643470 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.643393 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a27b7205-eaa0-4156-9c75-94d31a6d9eeb" (UID: "a27b7205-eaa0-4156-9c75-94d31a6d9eeb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:37:12.643470 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.643410 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-model-cache" (OuterVolumeSpecName: "model-cache") pod "a27b7205-eaa0-4156-9c75-94d31a6d9eeb" (UID: "a27b7205-eaa0-4156-9c75-94d31a6d9eeb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:37:12.643470 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.643422 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-home\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:37:12.645461 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.645436 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a27b7205-eaa0-4156-9c75-94d31a6d9eeb" (UID: "a27b7205-eaa0-4156-9c75-94d31a6d9eeb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:37:12.645461 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.645442 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-dshm" (OuterVolumeSpecName: "dshm") pod "a27b7205-eaa0-4156-9c75-94d31a6d9eeb" (UID: "a27b7205-eaa0-4156-9c75-94d31a6d9eeb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:37:12.645633 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.645439 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-kube-api-access-62lnc" (OuterVolumeSpecName: "kube-api-access-62lnc") pod "a27b7205-eaa0-4156-9c75-94d31a6d9eeb" (UID: "a27b7205-eaa0-4156-9c75-94d31a6d9eeb"). InnerVolumeSpecName "kube-api-access-62lnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:37:12.744352 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.744308 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62lnc\" (UniqueName: \"kubernetes.io/projected/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-kube-api-access-62lnc\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:37:12.744352 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.744345 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-tls-certs\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:37:12.744352 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.744357 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-model-cache\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:37:12.744352 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.744364 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-tmp-dir\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:37:12.744631 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:12.744372 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a27b7205-eaa0-4156-9c75-94d31a6d9eeb-dshm\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:37:13.286095 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.286021 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" podUID="252e3e6a-3ca9-413f-a884-ea0d618511ff" containerName="main" containerID="cri-o://97066b77fca0a75ad75d42a071575aa8acaed7a1216570a0d6aea3ed001b1cdd" gracePeriod=45 Jul 10 19:37:13.523435 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.523409 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" Jul 10 19:37:13.523435 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.523425 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb" event={"ID":"a27b7205-eaa0-4156-9c75-94d31a6d9eeb","Type":"ContainerDied","Data":"a7bec1980b54ff6b1a2d7b8492320bb9bd2653a860f438f5e0cde1c9c31a9adf"} Jul 10 19:37:13.523877 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.523478 2561 scope.go:117] "RemoveContainer" containerID="9de89fd1b345db1bebdb6a8e758b9a15dd99d75d29f25a1ff0014206a4c8246d" Jul 10 19:37:13.524874 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.524849 2561 generic.go:356] "Generic (PLEG): container finished" podID="252e3e6a-3ca9-413f-a884-ea0d618511ff" containerID="97066b77fca0a75ad75d42a071575aa8acaed7a1216570a0d6aea3ed001b1cdd" exitCode=0 Jul 10 19:37:13.524987 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.524886 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" event={"ID":"252e3e6a-3ca9-413f-a884-ea0d618511ff","Type":"ContainerDied","Data":"97066b77fca0a75ad75d42a071575aa8acaed7a1216570a0d6aea3ed001b1cdd"} Jul 10 19:37:13.524987 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.524911 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" event={"ID":"252e3e6a-3ca9-413f-a884-ea0d618511ff","Type":"ContainerDied","Data":"315303d4ea86d68bcd000737299cee30446cde7fe2c7e47af7f1890d25dbe023"} Jul 10 19:37:13.524987 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.524925 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="315303d4ea86d68bcd000737299cee30446cde7fe2c7e47af7f1890d25dbe023" Jul 10 19:37:13.529705 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.529684 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:37:13.564140 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.564050 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb"] Jul 10 19:37:13.569396 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.569366 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b1f450/auth-test-model-b1f450-kserve-7cc9d454dc-bphqb"] Jul 10 19:37:13.651430 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.651400 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-dshm\") pod \"252e3e6a-3ca9-413f-a884-ea0d618511ff\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " Jul 10 19:37:13.651604 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.651447 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-home\") pod \"252e3e6a-3ca9-413f-a884-ea0d618511ff\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " Jul 10 19:37:13.651604 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.651470 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-model-cache\") pod \"252e3e6a-3ca9-413f-a884-ea0d618511ff\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " Jul 10 19:37:13.651604 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.651488 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/252e3e6a-3ca9-413f-a884-ea0d618511ff-tls-certs\") pod \"252e3e6a-3ca9-413f-a884-ea0d618511ff\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " Jul 10 19:37:13.651604 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.651534 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlnpc\" (UniqueName: \"kubernetes.io/projected/252e3e6a-3ca9-413f-a884-ea0d618511ff-kube-api-access-nlnpc\") pod \"252e3e6a-3ca9-413f-a884-ea0d618511ff\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " Jul 10 19:37:13.651604 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.651572 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-tmp-dir\") pod \"252e3e6a-3ca9-413f-a884-ea0d618511ff\" (UID: \"252e3e6a-3ca9-413f-a884-ea0d618511ff\") " Jul 10 19:37:13.651870 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.651726 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-home" (OuterVolumeSpecName: "home") pod "252e3e6a-3ca9-413f-a884-ea0d618511ff" (UID: "252e3e6a-3ca9-413f-a884-ea0d618511ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:37:13.651870 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.651740 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "252e3e6a-3ca9-413f-a884-ea0d618511ff" (UID: "252e3e6a-3ca9-413f-a884-ea0d618511ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:37:13.652001 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.651878 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "252e3e6a-3ca9-413f-a884-ea0d618511ff" (UID: "252e3e6a-3ca9-413f-a884-ea0d618511ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:37:13.653840 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.653808 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-dshm" (OuterVolumeSpecName: "dshm") pod "252e3e6a-3ca9-413f-a884-ea0d618511ff" (UID: "252e3e6a-3ca9-413f-a884-ea0d618511ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:37:13.653967 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.653857 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/252e3e6a-3ca9-413f-a884-ea0d618511ff-kube-api-access-nlnpc" (OuterVolumeSpecName: "kube-api-access-nlnpc") pod "252e3e6a-3ca9-413f-a884-ea0d618511ff" (UID: "252e3e6a-3ca9-413f-a884-ea0d618511ff"). InnerVolumeSpecName "kube-api-access-nlnpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:37:13.653967 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.653950 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/252e3e6a-3ca9-413f-a884-ea0d618511ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "252e3e6a-3ca9-413f-a884-ea0d618511ff" (UID: "252e3e6a-3ca9-413f-a884-ea0d618511ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:37:13.752846 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.752802 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-model-cache\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:37:13.752846 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.752837 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/252e3e6a-3ca9-413f-a884-ea0d618511ff-tls-certs\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:37:13.752846 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.752846 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlnpc\" (UniqueName: \"kubernetes.io/projected/252e3e6a-3ca9-413f-a884-ea0d618511ff-kube-api-access-nlnpc\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:37:13.752846 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.752857 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-tmp-dir\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:37:13.753151 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.752866 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-dshm\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:37:13.753151 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.752873 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/252e3e6a-3ca9-413f-a884-ea0d618511ff-home\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:37:13.963782 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:13.963701 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a27b7205-eaa0-4156-9c75-94d31a6d9eeb" path="/var/lib/kubelet/pods/a27b7205-eaa0-4156-9c75-94d31a6d9eeb/volumes" Jul 10 19:37:14.530157 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:14.530129 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h" Jul 10 19:37:14.548754 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:14.548725 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h"] Jul 10 19:37:14.553223 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:14.553198 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b02417/auth-test-model-b02417-kserve-7f64b76655-pn97h"] Jul 10 19:37:15.962589 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:15.962556 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="252e3e6a-3ca9-413f-a884-ea0d618511ff" path="/var/lib/kubelet/pods/252e3e6a-3ca9-413f-a884-ea0d618511ff/volumes" Jul 10 19:37:16.608373 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.608333 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw"] Jul 10 19:37:16.608742 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.608723 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a27b7205-eaa0-4156-9c75-94d31a6d9eeb" containerName="main" Jul 10 19:37:16.608831 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.608745 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a27b7205-eaa0-4156-9c75-94d31a6d9eeb" containerName="main" Jul 10 19:37:16.608831 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.608758 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="252e3e6a-3ca9-413f-a884-ea0d618511ff" containerName="main" Jul 10 19:37:16.608831 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.608766 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="252e3e6a-3ca9-413f-a884-ea0d618511ff" containerName="main" Jul 10 19:37:16.609017 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.608835 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a27b7205-eaa0-4156-9c75-94d31a6d9eeb" containerName="main" Jul 10 19:37:16.609017 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.608851 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="252e3e6a-3ca9-413f-a884-ea0d618511ff" containerName="main" Jul 10 19:37:16.612913 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.612890 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:16.615736 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.615718 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b1f450\"/\"openshift-service-ca.crt\"" Jul 10 19:37:16.616671 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.616652 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-b1f450\"/\"kube-root-ca.crt\"" Jul 10 19:37:16.616765 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.616668 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b1f450\"/\"sub-test-model-b1f450-kserve-self-signed-certs\"" Jul 10 19:37:16.616765 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.616695 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b1f450\"/\"default-dockercfg-6gflp\"" Jul 10 19:37:16.623106 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.623076 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw"] Jul 10 19:37:16.693177 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.693114 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:16.777389 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.777353 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smg52\" (UniqueName: \"kubernetes.io/projected/5dee88e8-5296-4c40-a643-c1c7551c32df-kube-api-access-smg52\") pod \"sub-test-model-b1f450-kserve-86c9d56587-wprmw\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:16.777575 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.777418 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-dshm\") pod \"sub-test-model-b1f450-kserve-86c9d56587-wprmw\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:16.777575 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.777486 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-tmp-dir\") pod \"sub-test-model-b1f450-kserve-86c9d56587-wprmw\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:16.777575 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.777517 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-home\") pod \"sub-test-model-b1f450-kserve-86c9d56587-wprmw\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:16.777575 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.777535 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dee88e8-5296-4c40-a643-c1c7551c32df-tls-certs\") pod \"sub-test-model-b1f450-kserve-86c9d56587-wprmw\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:16.777575 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.777561 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-model-cache\") pod \"sub-test-model-b1f450-kserve-86c9d56587-wprmw\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:16.878694 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.878603 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-dshm\") pod \"sub-test-model-b1f450-kserve-86c9d56587-wprmw\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:16.878694 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.878638 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-tmp-dir\") pod \"sub-test-model-b1f450-kserve-86c9d56587-wprmw\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:16.878694 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.878658 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-home\") pod \"sub-test-model-b1f450-kserve-86c9d56587-wprmw\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:16.878694 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.878674 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dee88e8-5296-4c40-a643-c1c7551c32df-tls-certs\") pod \"sub-test-model-b1f450-kserve-86c9d56587-wprmw\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:16.878694 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.878695 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-model-cache\") pod \"sub-test-model-b1f450-kserve-86c9d56587-wprmw\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:16.879129 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.878733 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smg52\" (UniqueName: \"kubernetes.io/projected/5dee88e8-5296-4c40-a643-c1c7551c32df-kube-api-access-smg52\") pod \"sub-test-model-b1f450-kserve-86c9d56587-wprmw\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:16.879129 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.879085 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-home\") pod \"sub-test-model-b1f450-kserve-86c9d56587-wprmw\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:16.879224 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.879148 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-tmp-dir\") pod \"sub-test-model-b1f450-kserve-86c9d56587-wprmw\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:16.879224 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.879172 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-model-cache\") pod \"sub-test-model-b1f450-kserve-86c9d56587-wprmw\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:16.881189 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.881168 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-dshm\") pod \"sub-test-model-b1f450-kserve-86c9d56587-wprmw\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:16.881591 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.881572 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dee88e8-5296-4c40-a643-c1c7551c32df-tls-certs\") pod \"sub-test-model-b1f450-kserve-86c9d56587-wprmw\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:16.888086 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.888064 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smg52\" (UniqueName: \"kubernetes.io/projected/5dee88e8-5296-4c40-a643-c1c7551c32df-kube-api-access-smg52\") pod \"sub-test-model-b1f450-kserve-86c9d56587-wprmw\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:16.922241 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:16.922202 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:17.059107 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:17.059074 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw"] Jul 10 19:37:17.061844 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:37:17.061817 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5dee88e8_5296_4c40_a643_c1c7551c32df.slice/crio-9a8762a61c7a56216c8a37e0c6de103bb54a3aa63bc313a0dd8903e5d2a2c7e1 WatchSource:0}: Error finding container 9a8762a61c7a56216c8a37e0c6de103bb54a3aa63bc313a0dd8903e5d2a2c7e1: Status 404 returned error can't find the container with id 9a8762a61c7a56216c8a37e0c6de103bb54a3aa63bc313a0dd8903e5d2a2c7e1 Jul 10 19:37:17.548912 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:17.548875 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" event={"ID":"5dee88e8-5296-4c40-a643-c1c7551c32df","Type":"ContainerStarted","Data":"6bbe5bb014d0827b6bcf7630abc5accaaacc58b8d9d517b47f0215fc1680f18f"} Jul 10 19:37:17.548912 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:17.548917 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" event={"ID":"5dee88e8-5296-4c40-a643-c1c7551c32df","Type":"ContainerStarted","Data":"9a8762a61c7a56216c8a37e0c6de103bb54a3aa63bc313a0dd8903e5d2a2c7e1"} Jul 10 19:37:17.569527 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:17.569473 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" podStartSLOduration=1.569457652 podStartE2EDuration="1.569457652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:37:17.568614835 +0000 UTC m=+2204.169756971" watchObservedRunningTime="2026-07-10 19:37:17.569457652 +0000 UTC m=+2204.170599786" Jul 10 19:37:21.500299 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:21.500260 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:37:26.922663 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:26.922622 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:26.923195 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:26.922684 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:26.935203 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:26.935174 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:27.619637 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:27.619607 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:37:32.081858 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.081826 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv"] Jul 10 19:37:32.085282 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.085263 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.088206 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.088184 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b02417\"/\"default-dockercfg-nflsl\"" Jul 10 19:37:32.089080 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.089060 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b02417\"/\"kube-root-ca.crt\"" Jul 10 19:37:32.089186 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.089114 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b02417\"/\"openshift-service-ca.crt\"" Jul 10 19:37:32.089186 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.089170 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b02417\"/\"sub-test-model-b02417-kserve-self-signed-certs\"" Jul 10 19:37:32.097185 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.097163 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv"] Jul 10 19:37:32.100473 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.100452 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b8x6\" (UniqueName: \"kubernetes.io/projected/ff2a974a-63c7-4776-94c5-94f19047c5af-kube-api-access-5b8x6\") pod \"sub-test-model-b02417-kserve-5cd4654755-j28cv\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.100559 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.100484 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff2a974a-63c7-4776-94c5-94f19047c5af-tls-certs\") pod \"sub-test-model-b02417-kserve-5cd4654755-j28cv\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.100559 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.100508 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-dshm\") pod \"sub-test-model-b02417-kserve-5cd4654755-j28cv\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.100634 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.100559 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-model-cache\") pod \"sub-test-model-b02417-kserve-5cd4654755-j28cv\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.100634 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.100607 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-tmp-dir\") pod \"sub-test-model-b02417-kserve-5cd4654755-j28cv\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.100700 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.100683 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-home\") pod \"sub-test-model-b02417-kserve-5cd4654755-j28cv\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.126280 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.126249 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:37:32.201198 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.201158 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-home\") pod \"sub-test-model-b02417-kserve-5cd4654755-j28cv\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.201198 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.201201 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b8x6\" (UniqueName: \"kubernetes.io/projected/ff2a974a-63c7-4776-94c5-94f19047c5af-kube-api-access-5b8x6\") pod \"sub-test-model-b02417-kserve-5cd4654755-j28cv\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.201399 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.201319 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff2a974a-63c7-4776-94c5-94f19047c5af-tls-certs\") pod \"sub-test-model-b02417-kserve-5cd4654755-j28cv\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.201399 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.201376 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-dshm\") pod \"sub-test-model-b02417-kserve-5cd4654755-j28cv\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.201502 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.201487 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-model-cache\") pod \"sub-test-model-b02417-kserve-5cd4654755-j28cv\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.201538 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.201521 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-tmp-dir\") pod \"sub-test-model-b02417-kserve-5cd4654755-j28cv\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.201628 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.201594 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-home\") pod \"sub-test-model-b02417-kserve-5cd4654755-j28cv\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.201823 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.201802 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-model-cache\") pod \"sub-test-model-b02417-kserve-5cd4654755-j28cv\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.201905 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.201841 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-tmp-dir\") pod \"sub-test-model-b02417-kserve-5cd4654755-j28cv\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.203736 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.203714 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-dshm\") pod \"sub-test-model-b02417-kserve-5cd4654755-j28cv\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.203899 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.203881 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff2a974a-63c7-4776-94c5-94f19047c5af-tls-certs\") pod \"sub-test-model-b02417-kserve-5cd4654755-j28cv\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.210098 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.210076 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b8x6\" (UniqueName: \"kubernetes.io/projected/ff2a974a-63c7-4776-94c5-94f19047c5af-kube-api-access-5b8x6\") pod \"sub-test-model-b02417-kserve-5cd4654755-j28cv\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.395293 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.395203 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:32.536793 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.536766 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv"] Jul 10 19:37:32.539486 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:37:32.539458 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff2a974a_63c7_4776_94c5_94f19047c5af.slice/crio-f8ab1f14b70058d0b1b84f2401f6ab5c05b2c19cd157169f2a052da9b0315f96 WatchSource:0}: Error finding container f8ab1f14b70058d0b1b84f2401f6ab5c05b2c19cd157169f2a052da9b0315f96: Status 404 returned error can't find the container with id f8ab1f14b70058d0b1b84f2401f6ab5c05b2c19cd157169f2a052da9b0315f96 Jul 10 19:37:32.645968 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.645831 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" event={"ID":"ff2a974a-63c7-4776-94c5-94f19047c5af","Type":"ContainerStarted","Data":"610303e7e3436a5c4f4cb59093405f003b1c6e297d374dfb25f84aafb48f30fe"} Jul 10 19:37:32.645968 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.645883 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" event={"ID":"ff2a974a-63c7-4776-94c5-94f19047c5af","Type":"ContainerStarted","Data":"f8ab1f14b70058d0b1b84f2401f6ab5c05b2c19cd157169f2a052da9b0315f96"} Jul 10 19:37:32.665043 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:32.664994 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" podStartSLOduration=0.664978744 podStartE2EDuration="664.978744ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:37:32.663389005 +0000 UTC m=+2219.264531141" watchObservedRunningTime="2026-07-10 19:37:32.664978744 +0000 UTC m=+2219.266120857" Jul 10 19:37:37.988114 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:37.988080 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:37:42.396334 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:42.396294 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:42.396727 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:42.396348 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:42.408753 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:42.408726 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:42.709752 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:42.709656 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:37:59.100298 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:37:59.100223 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:38:14.844637 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:14.844599 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:38:21.782207 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:21.782173 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:21.889109 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:21.889065 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:22.179415 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:22.179328 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw"] Jul 10 19:38:22.193571 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:22.193510 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:23.109005 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:23.108962 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv"] Jul 10 19:38:23.131243 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:23.131210 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:29.985245 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:29.985202 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:38:31.696874 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.696837 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld"] Jul 10 19:38:31.700422 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.700397 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:31.702970 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.702946 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-b1f450\"/\"rate-test-model-b1f450-kserve-self-signed-certs\"" Jul 10 19:38:31.703620 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.703598 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-home\") pod \"rate-test-model-b1f450-kserve-644487f7f5-79zld\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:31.703765 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.703639 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-dshm\") pod \"rate-test-model-b1f450-kserve-644487f7f5-79zld\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:31.703765 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.703655 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-tmp-dir\") pod \"rate-test-model-b1f450-kserve-644487f7f5-79zld\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:31.703765 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.703739 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-model-cache\") pod \"rate-test-model-b1f450-kserve-644487f7f5-79zld\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:31.703950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.703815 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmtr4\" (UniqueName: \"kubernetes.io/projected/a2faff33-7cc3-4ab5-85c1-238a39dc0861-kube-api-access-kmtr4\") pod \"rate-test-model-b1f450-kserve-644487f7f5-79zld\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:31.703950 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.703868 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2faff33-7cc3-4ab5-85c1-238a39dc0861-tls-certs\") pod \"rate-test-model-b1f450-kserve-644487f7f5-79zld\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:31.711011 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.710985 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld"] Jul 10 19:38:31.734208 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.734163 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:31.804842 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.804808 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2faff33-7cc3-4ab5-85c1-238a39dc0861-tls-certs\") pod \"rate-test-model-b1f450-kserve-644487f7f5-79zld\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:31.805062 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.804860 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-home\") pod \"rate-test-model-b1f450-kserve-644487f7f5-79zld\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:31.805062 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.804903 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-dshm\") pod \"rate-test-model-b1f450-kserve-644487f7f5-79zld\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:31.805062 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.804925 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-tmp-dir\") pod \"rate-test-model-b1f450-kserve-644487f7f5-79zld\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:31.805062 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.804985 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-model-cache\") pod \"rate-test-model-b1f450-kserve-644487f7f5-79zld\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:31.805062 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.805027 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmtr4\" (UniqueName: \"kubernetes.io/projected/a2faff33-7cc3-4ab5-85c1-238a39dc0861-kube-api-access-kmtr4\") pod \"rate-test-model-b1f450-kserve-644487f7f5-79zld\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:31.805476 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.805453 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-home\") pod \"rate-test-model-b1f450-kserve-644487f7f5-79zld\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:31.805584 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.805489 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-tmp-dir\") pod \"rate-test-model-b1f450-kserve-644487f7f5-79zld\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:31.805584 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.805518 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-model-cache\") pod \"rate-test-model-b1f450-kserve-644487f7f5-79zld\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:31.807352 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.807328 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-dshm\") pod \"rate-test-model-b1f450-kserve-644487f7f5-79zld\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:31.807609 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.807593 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2faff33-7cc3-4ab5-85c1-238a39dc0861-tls-certs\") pod \"rate-test-model-b1f450-kserve-644487f7f5-79zld\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:31.816984 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:31.816959 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmtr4\" (UniqueName: \"kubernetes.io/projected/a2faff33-7cc3-4ab5-85c1-238a39dc0861-kube-api-access-kmtr4\") pod \"rate-test-model-b1f450-kserve-644487f7f5-79zld\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:32.010737 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:32.010703 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:32.147472 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:32.147447 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld"] Jul 10 19:38:32.149254 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:38:32.149227 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2faff33_7cc3_4ab5_85c1_238a39dc0861.slice/crio-446cdd22940697c73c8868ec87e8438db704a921311baf291a46ef45018d8147 WatchSource:0}: Error finding container 446cdd22940697c73c8868ec87e8438db704a921311baf291a46ef45018d8147: Status 404 returned error can't find the container with id 446cdd22940697c73c8868ec87e8438db704a921311baf291a46ef45018d8147 Jul 10 19:38:32.954482 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:32.954445 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" event={"ID":"a2faff33-7cc3-4ab5-85c1-238a39dc0861","Type":"ContainerStarted","Data":"7a883f45364cbe599614cd58c58f23b24f8d9804b429a09e3f9a503588caf1d6"} Jul 10 19:38:32.955005 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:32.954490 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" event={"ID":"a2faff33-7cc3-4ab5-85c1-238a39dc0861","Type":"ContainerStarted","Data":"446cdd22940697c73c8868ec87e8438db704a921311baf291a46ef45018d8147"} Jul 10 19:38:32.973443 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:32.973391 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" podStartSLOduration=1.973376543 podStartE2EDuration="1.973376543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:38:32.971650652 +0000 UTC m=+2279.572792789" watchObservedRunningTime="2026-07-10 19:38:32.973376543 +0000 UTC m=+2279.574518727" Jul 10 19:38:37.200485 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.200438 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" podUID="5dee88e8-5296-4c40-a643-c1c7551c32df" containerName="main" containerID="cri-o://6bbe5bb014d0827b6bcf7630abc5accaaacc58b8d9d517b47f0215fc1680f18f" gracePeriod=45 Jul 10 19:38:37.451712 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.451630 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:38:37.549964 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.549910 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smg52\" (UniqueName: \"kubernetes.io/projected/5dee88e8-5296-4c40-a643-c1c7551c32df-kube-api-access-smg52\") pod \"5dee88e8-5296-4c40-a643-c1c7551c32df\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " Jul 10 19:38:37.549964 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.549960 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-model-cache\") pod \"5dee88e8-5296-4c40-a643-c1c7551c32df\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " Jul 10 19:38:37.550230 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.549981 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-tmp-dir\") pod \"5dee88e8-5296-4c40-a643-c1c7551c32df\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " Jul 10 19:38:37.550230 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.550003 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-dshm\") pod \"5dee88e8-5296-4c40-a643-c1c7551c32df\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " Jul 10 19:38:37.550230 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.550021 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-home\") pod \"5dee88e8-5296-4c40-a643-c1c7551c32df\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " Jul 10 19:38:37.550230 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.550044 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dee88e8-5296-4c40-a643-c1c7551c32df-tls-certs\") pod \"5dee88e8-5296-4c40-a643-c1c7551c32df\" (UID: \"5dee88e8-5296-4c40-a643-c1c7551c32df\") " Jul 10 19:38:37.550387 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.550256 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-model-cache" (OuterVolumeSpecName: "model-cache") pod "5dee88e8-5296-4c40-a643-c1c7551c32df" (UID: "5dee88e8-5296-4c40-a643-c1c7551c32df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:38:37.550387 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.550270 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5dee88e8-5296-4c40-a643-c1c7551c32df" (UID: "5dee88e8-5296-4c40-a643-c1c7551c32df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:38:37.550387 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.550290 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-home" (OuterVolumeSpecName: "home") pod "5dee88e8-5296-4c40-a643-c1c7551c32df" (UID: "5dee88e8-5296-4c40-a643-c1c7551c32df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:38:37.552244 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.552224 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-dshm" (OuterVolumeSpecName: "dshm") pod "5dee88e8-5296-4c40-a643-c1c7551c32df" (UID: "5dee88e8-5296-4c40-a643-c1c7551c32df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:38:37.552309 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.552294 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dee88e8-5296-4c40-a643-c1c7551c32df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5dee88e8-5296-4c40-a643-c1c7551c32df" (UID: "5dee88e8-5296-4c40-a643-c1c7551c32df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:38:37.552393 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.552379 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dee88e8-5296-4c40-a643-c1c7551c32df-kube-api-access-smg52" (OuterVolumeSpecName: "kube-api-access-smg52") pod "5dee88e8-5296-4c40-a643-c1c7551c32df" (UID: "5dee88e8-5296-4c40-a643-c1c7551c32df"). InnerVolumeSpecName "kube-api-access-smg52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:38:37.651248 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.651212 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smg52\" (UniqueName: \"kubernetes.io/projected/5dee88e8-5296-4c40-a643-c1c7551c32df-kube-api-access-smg52\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:38:37.651248 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.651240 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-model-cache\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:38:37.651248 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.651250 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-tmp-dir\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:38:37.651248 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.651258 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-dshm\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:38:37.651507 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.651265 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dee88e8-5296-4c40-a643-c1c7551c32df-home\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:38:37.651507 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.651272 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dee88e8-5296-4c40-a643-c1c7551c32df-tls-certs\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:38:37.980523 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.980487 2561 generic.go:356] "Generic (PLEG): container finished" podID="5dee88e8-5296-4c40-a643-c1c7551c32df" containerID="6bbe5bb014d0827b6bcf7630abc5accaaacc58b8d9d517b47f0215fc1680f18f" exitCode=0 Jul 10 19:38:37.980676 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.980534 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" Jul 10 19:38:37.980676 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.980564 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" event={"ID":"5dee88e8-5296-4c40-a643-c1c7551c32df","Type":"ContainerDied","Data":"6bbe5bb014d0827b6bcf7630abc5accaaacc58b8d9d517b47f0215fc1680f18f"} Jul 10 19:38:37.980676 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.980606 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw" event={"ID":"5dee88e8-5296-4c40-a643-c1c7551c32df","Type":"ContainerDied","Data":"9a8762a61c7a56216c8a37e0c6de103bb54a3aa63bc313a0dd8903e5d2a2c7e1"} Jul 10 19:38:37.980676 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.980624 2561 scope.go:117] "RemoveContainer" containerID="6bbe5bb014d0827b6bcf7630abc5accaaacc58b8d9d517b47f0215fc1680f18f" Jul 10 19:38:37.989384 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.989364 2561 scope.go:117] "RemoveContainer" containerID="6bbe5bb014d0827b6bcf7630abc5accaaacc58b8d9d517b47f0215fc1680f18f" Jul 10 19:38:37.989701 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:38:37.989680 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bbe5bb014d0827b6bcf7630abc5accaaacc58b8d9d517b47f0215fc1680f18f\": container with ID starting with 6bbe5bb014d0827b6bcf7630abc5accaaacc58b8d9d517b47f0215fc1680f18f not found: ID does not exist" containerID="6bbe5bb014d0827b6bcf7630abc5accaaacc58b8d9d517b47f0215fc1680f18f" Jul 10 19:38:37.989785 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.989708 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bbe5bb014d0827b6bcf7630abc5accaaacc58b8d9d517b47f0215fc1680f18f"} err="failed to get container status \"6bbe5bb014d0827b6bcf7630abc5accaaacc58b8d9d517b47f0215fc1680f18f\": rpc error: code = NotFound desc = could not find container \"6bbe5bb014d0827b6bcf7630abc5accaaacc58b8d9d517b47f0215fc1680f18f\": container with ID starting with 6bbe5bb014d0827b6bcf7630abc5accaaacc58b8d9d517b47f0215fc1680f18f not found: ID does not exist" Jul 10 19:38:37.998096 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:37.998058 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw"] Jul 10 19:38:38.000052 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.000030 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b1f450/sub-test-model-b1f450-kserve-86c9d56587-wprmw"] Jul 10 19:38:38.129388 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.129321 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" podUID="ff2a974a-63c7-4776-94c5-94f19047c5af" containerName="main" containerID="cri-o://610303e7e3436a5c4f4cb59093405f003b1c6e297d374dfb25f84aafb48f30fe" gracePeriod=45 Jul 10 19:38:38.378298 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.378269 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:38:38.458735 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.458708 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-home\") pod \"ff2a974a-63c7-4776-94c5-94f19047c5af\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " Jul 10 19:38:38.458924 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.458761 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff2a974a-63c7-4776-94c5-94f19047c5af-tls-certs\") pod \"ff2a974a-63c7-4776-94c5-94f19047c5af\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " Jul 10 19:38:38.458924 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.458787 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5b8x6\" (UniqueName: \"kubernetes.io/projected/ff2a974a-63c7-4776-94c5-94f19047c5af-kube-api-access-5b8x6\") pod \"ff2a974a-63c7-4776-94c5-94f19047c5af\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " Jul 10 19:38:38.458924 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.458808 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-dshm\") pod \"ff2a974a-63c7-4776-94c5-94f19047c5af\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " Jul 10 19:38:38.458924 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.458830 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-model-cache\") pod \"ff2a974a-63c7-4776-94c5-94f19047c5af\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " Jul 10 19:38:38.458924 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.458856 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-tmp-dir\") pod \"ff2a974a-63c7-4776-94c5-94f19047c5af\" (UID: \"ff2a974a-63c7-4776-94c5-94f19047c5af\") " Jul 10 19:38:38.459240 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.459052 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-home" (OuterVolumeSpecName: "home") pod "ff2a974a-63c7-4776-94c5-94f19047c5af" (UID: "ff2a974a-63c7-4776-94c5-94f19047c5af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:38:38.459240 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.459170 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff2a974a-63c7-4776-94c5-94f19047c5af" (UID: "ff2a974a-63c7-4776-94c5-94f19047c5af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:38:38.459240 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.459198 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff2a974a-63c7-4776-94c5-94f19047c5af" (UID: "ff2a974a-63c7-4776-94c5-94f19047c5af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:38:38.461056 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.461027 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-dshm" (OuterVolumeSpecName: "dshm") pod "ff2a974a-63c7-4776-94c5-94f19047c5af" (UID: "ff2a974a-63c7-4776-94c5-94f19047c5af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:38:38.461056 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.461027 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff2a974a-63c7-4776-94c5-94f19047c5af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff2a974a-63c7-4776-94c5-94f19047c5af" (UID: "ff2a974a-63c7-4776-94c5-94f19047c5af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:38:38.461200 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.461108 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2a974a-63c7-4776-94c5-94f19047c5af-kube-api-access-5b8x6" (OuterVolumeSpecName: "kube-api-access-5b8x6") pod "ff2a974a-63c7-4776-94c5-94f19047c5af" (UID: "ff2a974a-63c7-4776-94c5-94f19047c5af"). InnerVolumeSpecName "kube-api-access-5b8x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:38:38.560298 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.560205 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff2a974a-63c7-4776-94c5-94f19047c5af-tls-certs\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:38:38.560298 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.560237 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b8x6\" (UniqueName: \"kubernetes.io/projected/ff2a974a-63c7-4776-94c5-94f19047c5af-kube-api-access-5b8x6\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:38:38.560298 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.560246 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-dshm\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:38:38.560298 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.560257 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-model-cache\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:38:38.560298 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.560264 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-tmp-dir\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:38:38.560298 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.560271 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff2a974a-63c7-4776-94c5-94f19047c5af-home\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:38:38.988079 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.987988 2561 generic.go:356] "Generic (PLEG): container finished" podID="ff2a974a-63c7-4776-94c5-94f19047c5af" containerID="610303e7e3436a5c4f4cb59093405f003b1c6e297d374dfb25f84aafb48f30fe" exitCode=0 Jul 10 19:38:38.988079 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.988032 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" Jul 10 19:38:38.988287 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.988084 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" event={"ID":"ff2a974a-63c7-4776-94c5-94f19047c5af","Type":"ContainerDied","Data":"610303e7e3436a5c4f4cb59093405f003b1c6e297d374dfb25f84aafb48f30fe"} Jul 10 19:38:38.988287 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.988119 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv" event={"ID":"ff2a974a-63c7-4776-94c5-94f19047c5af","Type":"ContainerDied","Data":"f8ab1f14b70058d0b1b84f2401f6ab5c05b2c19cd157169f2a052da9b0315f96"} Jul 10 19:38:38.988287 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.988138 2561 scope.go:117] "RemoveContainer" containerID="610303e7e3436a5c4f4cb59093405f003b1c6e297d374dfb25f84aafb48f30fe" Jul 10 19:38:38.997300 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.997284 2561 scope.go:117] "RemoveContainer" containerID="610303e7e3436a5c4f4cb59093405f003b1c6e297d374dfb25f84aafb48f30fe" Jul 10 19:38:38.997538 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:38:38.997516 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"610303e7e3436a5c4f4cb59093405f003b1c6e297d374dfb25f84aafb48f30fe\": container with ID starting with 610303e7e3436a5c4f4cb59093405f003b1c6e297d374dfb25f84aafb48f30fe not found: ID does not exist" containerID="610303e7e3436a5c4f4cb59093405f003b1c6e297d374dfb25f84aafb48f30fe" Jul 10 19:38:38.997575 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:38.997548 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"610303e7e3436a5c4f4cb59093405f003b1c6e297d374dfb25f84aafb48f30fe"} err="failed to get container status \"610303e7e3436a5c4f4cb59093405f003b1c6e297d374dfb25f84aafb48f30fe\": rpc error: code = NotFound desc = could not find container \"610303e7e3436a5c4f4cb59093405f003b1c6e297d374dfb25f84aafb48f30fe\": container with ID starting with 610303e7e3436a5c4f4cb59093405f003b1c6e297d374dfb25f84aafb48f30fe not found: ID does not exist" Jul 10 19:38:39.009393 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:39.009363 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv"] Jul 10 19:38:39.013617 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:39.013593 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b02417/sub-test-model-b02417-kserve-5cd4654755-j28cv"] Jul 10 19:38:39.963914 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:39.963873 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dee88e8-5296-4c40-a643-c1c7551c32df" path="/var/lib/kubelet/pods/5dee88e8-5296-4c40-a643-c1c7551c32df/volumes" Jul 10 19:38:39.964471 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:39.964449 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff2a974a-63c7-4776-94c5-94f19047c5af" path="/var/lib/kubelet/pods/ff2a974a-63c7-4776-94c5-94f19047c5af/volumes" Jul 10 19:38:41.889421 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:41.889382 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:38:42.011067 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:42.011014 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:42.011067 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:42.011068 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:42.023316 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:42.023281 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:43.020347 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:43.020314 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:38:54.894393 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.894358 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq"] Jul 10 19:38:54.894766 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.894679 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff2a974a-63c7-4776-94c5-94f19047c5af" containerName="main" Jul 10 19:38:54.894766 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.894692 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff2a974a-63c7-4776-94c5-94f19047c5af" containerName="main" Jul 10 19:38:54.894766 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.894718 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dee88e8-5296-4c40-a643-c1c7551c32df" containerName="main" Jul 10 19:38:54.894766 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.894724 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dee88e8-5296-4c40-a643-c1c7551c32df" containerName="main" Jul 10 19:38:54.894964 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.894773 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff2a974a-63c7-4776-94c5-94f19047c5af" containerName="main" Jul 10 19:38:54.894964 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.894784 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dee88e8-5296-4c40-a643-c1c7551c32df" containerName="main" Jul 10 19:38:54.898994 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.898974 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:54.902611 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.902584 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b02417\"/\"rate-test-model-b02417-kserve-self-signed-certs\"" Jul 10 19:38:54.902611 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.902605 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b02417\"/\"kube-root-ca.crt\"" Jul 10 19:38:54.902793 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.902586 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b02417\"/\"default-dockercfg-nflsl\"" Jul 10 19:38:54.902793 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.902585 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b02417\"/\"openshift-service-ca.crt\"" Jul 10 19:38:54.909017 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.908994 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq"] Jul 10 19:38:54.925570 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.925534 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:38:54.999023 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.998982 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-tmp-dir\") pod \"rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:54.999205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.999063 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-model-cache\") pod \"rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:54.999205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.999088 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b75041e-6e3a-49dd-81e6-9232008743e8-tls-certs\") pod \"rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:54.999205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.999110 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-home\") pod \"rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:54.999205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.999137 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-dshm\") pod \"rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:54.999205 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:54.999169 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzw9x\" (UniqueName: \"kubernetes.io/projected/9b75041e-6e3a-49dd-81e6-9232008743e8-kube-api-access-jzw9x\") pod \"rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:55.100129 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:55.100084 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-home\") pod \"rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:55.100129 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:55.100129 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-dshm\") pod \"rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:55.100397 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:55.100146 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzw9x\" (UniqueName: \"kubernetes.io/projected/9b75041e-6e3a-49dd-81e6-9232008743e8-kube-api-access-jzw9x\") pod \"rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:55.100397 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:55.100193 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-tmp-dir\") pod \"rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:55.100397 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:55.100245 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-model-cache\") pod \"rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:55.100397 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:55.100271 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b75041e-6e3a-49dd-81e6-9232008743e8-tls-certs\") pod \"rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:55.100611 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:55.100583 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-home\") pod \"rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:55.100657 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:55.100606 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-tmp-dir\") pod \"rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:55.100692 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:55.100663 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-model-cache\") pod \"rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:55.102568 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:55.102538 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-dshm\") pod \"rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:55.102840 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:55.102821 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b75041e-6e3a-49dd-81e6-9232008743e8-tls-certs\") pod \"rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:55.109244 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:55.109222 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzw9x\" (UniqueName: \"kubernetes.io/projected/9b75041e-6e3a-49dd-81e6-9232008743e8-kube-api-access-jzw9x\") pod \"rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:55.208529 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:55.208437 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:38:55.344430 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:55.344335 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq"] Jul 10 19:38:55.346662 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:38:55.346634 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b75041e_6e3a_49dd_81e6_9232008743e8.slice/crio-59e48857c40b1de867cbe22822041cecbe61e6e8843bdd848dbd76316cde41f0 WatchSource:0}: Error finding container 59e48857c40b1de867cbe22822041cecbe61e6e8843bdd848dbd76316cde41f0: Status 404 returned error can't find the container with id 59e48857c40b1de867cbe22822041cecbe61e6e8843bdd848dbd76316cde41f0 Jul 10 19:38:56.082175 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:56.082135 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" event={"ID":"9b75041e-6e3a-49dd-81e6-9232008743e8","Type":"ContainerStarted","Data":"c49fb1789fc9459cf154fa200eeb6517159ef9827ef80d0fe2ec3ffb3a022893"} Jul 10 19:38:56.082175 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:56.082181 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" event={"ID":"9b75041e-6e3a-49dd-81e6-9232008743e8","Type":"ContainerStarted","Data":"59e48857c40b1de867cbe22822041cecbe61e6e8843bdd848dbd76316cde41f0"} Jul 10 19:38:56.104544 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:56.104489 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" podStartSLOduration=2.104470026 podStartE2EDuration="2.104470026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:38:56.10139658 +0000 UTC m=+2302.702538706" watchObservedRunningTime="2026-07-10 19:38:56.104470026 +0000 UTC m=+2302.705612163" Jul 10 19:38:59.000454 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:38:59.000418 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:39:03.569663 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:03.569616 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:05.208920 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:05.208882 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:39:05.209348 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:05.208969 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:39:05.221428 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:05.221394 2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:39:06.141860 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:06.141828 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:39:18.496409 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:18.496368 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:39:20.930306 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:20.930270 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:21.031001 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:21.030930 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:29.220987 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:29.220931 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:29.308477 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:29.308428 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:32.890008 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:32.889972 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:39:46.640957 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:46.640909 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:39:50.733360 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:50.733317 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld"] Jul 10 19:39:50.752856 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:50.752817 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:51.598050 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:51.598018 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq"] Jul 10 19:39:51.617418 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:51.617381 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:39:56.060567 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:56.060531 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7655-140463795807024"] Jul 10 19:39:56.063817 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:56.063800 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7655-140463795807024" Jul 10 19:39:56.066280 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:56.066257 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-q7hmp\"" Jul 10 19:39:56.071843 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:56.071821 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7655-140463795807024"] Jul 10 19:39:56.135196 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:56.135157 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6572p\" (UniqueName: \"kubernetes.io/projected/731b000c-32eb-476b-868b-ce18f5d9c302-kube-api-access-6572p\") pod \"test-curl-7655-140463795807024\" (UID: \"731b000c-32eb-476b-868b-ce18f5d9c302\") " pod="openshift-ingress/test-curl-7655-140463795807024" Jul 10 19:39:56.236285 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:56.236240 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6572p\" (UniqueName: \"kubernetes.io/projected/731b000c-32eb-476b-868b-ce18f5d9c302-kube-api-access-6572p\") pod \"test-curl-7655-140463795807024\" (UID: \"731b000c-32eb-476b-868b-ce18f5d9c302\") " pod="openshift-ingress/test-curl-7655-140463795807024" Jul 10 19:39:56.245451 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:56.245422 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6572p\" (UniqueName: \"kubernetes.io/projected/731b000c-32eb-476b-868b-ce18f5d9c302-kube-api-access-6572p\") pod \"test-curl-7655-140463795807024\" (UID: \"731b000c-32eb-476b-868b-ce18f5d9c302\") " pod="openshift-ingress/test-curl-7655-140463795807024" Jul 10 19:39:56.374245 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:56.374153 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7655-140463795807024" Jul 10 19:39:56.507909 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:56.507886 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7655-140463795807024"] Jul 10 19:39:56.510179 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:39:56.510148 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod731b000c_32eb_476b_868b_ce18f5d9c302.slice/crio-4a52c7736acd4d1a4ad10dd8cb346e37b919ed8e3228403a45c330a95e91e17b WatchSource:0}: Error finding container 4a52c7736acd4d1a4ad10dd8cb346e37b919ed8e3228403a45c330a95e91e17b: Status 404 returned error can't find the container with id 4a52c7736acd4d1a4ad10dd8cb346e37b919ed8e3228403a45c330a95e91e17b Jul 10 19:39:56.511660 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:56.511638 2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 19:39:57.391728 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:57.391695 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7655-140463795807024" event={"ID":"731b000c-32eb-476b-868b-ce18f5d9c302","Type":"ContainerStarted","Data":"169c2c468e2d382534a051dddb060b9444cab186f83e5981b7a97dc90ccb5fca"} Jul 10 19:39:57.392188 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:57.391737 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7655-140463795807024" event={"ID":"731b000c-32eb-476b-868b-ce18f5d9c302","Type":"ContainerStarted","Data":"4a52c7736acd4d1a4ad10dd8cb346e37b919ed8e3228403a45c330a95e91e17b"} Jul 10 19:39:57.409008 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:57.408921 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/test-curl-7655-140463795807024" podStartSLOduration=0.656161534 podStartE2EDuration="1.408905003s" totalImagesPullingTime="752.743469ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:39:57.40736352 +0000 UTC m=+2364.008505679" watchObservedRunningTime="2026-07-10 19:39:57.408905003 +0000 UTC m=+2364.010047116" Jul 10 19:39:57.518607 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:57.518534 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7655-140463795807024_731b000c-32eb-476b-868b-ce18f5d9c302/test-curl-7655-140463795807024/0.log" Jul 10 19:39:58.398030 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:58.397996 2561 generic.go:356] "Generic (PLEG): container finished" podID="731b000c-32eb-476b-868b-ce18f5d9c302" containerID="169c2c468e2d382534a051dddb060b9444cab186f83e5981b7a97dc90ccb5fca" exitCode=0 Jul 10 19:39:58.398409 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:58.398040 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7655-140463795807024" event={"ID":"731b000c-32eb-476b-868b-ce18f5d9c302","Type":"ContainerDied","Data":"169c2c468e2d382534a051dddb060b9444cab186f83e5981b7a97dc90ccb5fca"} Jul 10 19:39:59.546569 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:59.546542 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7655-140463795807024" Jul 10 19:39:59.579385 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:59.579351 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7655-140463795807024"] Jul 10 19:39:59.584213 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:59.584187 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7655-140463795807024"] Jul 10 19:39:59.667711 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:59.667626 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6572p\" (UniqueName: \"kubernetes.io/projected/731b000c-32eb-476b-868b-ce18f5d9c302-kube-api-access-6572p\") pod \"731b000c-32eb-476b-868b-ce18f5d9c302\" (UID: \"731b000c-32eb-476b-868b-ce18f5d9c302\") " Jul 10 19:39:59.669960 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:59.669908 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731b000c-32eb-476b-868b-ce18f5d9c302-kube-api-access-6572p" (OuterVolumeSpecName: "kube-api-access-6572p") pod "731b000c-32eb-476b-868b-ce18f5d9c302" (UID: "731b000c-32eb-476b-868b-ce18f5d9c302"). InnerVolumeSpecName "kube-api-access-6572p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:39:59.739284 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:59.739244 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7655-140463795698688"] Jul 10 19:39:59.739706 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:59.739651 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="731b000c-32eb-476b-868b-ce18f5d9c302" containerName="test-curl-7655-140463795807024" Jul 10 19:39:59.739706 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:59.739672 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="731b000c-32eb-476b-868b-ce18f5d9c302" containerName="test-curl-7655-140463795807024" Jul 10 19:39:59.739851 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:59.739794 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="731b000c-32eb-476b-868b-ce18f5d9c302" containerName="test-curl-7655-140463795807024" Jul 10 19:39:59.742759 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:59.742737 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7655-140463795698688" Jul 10 19:39:59.750840 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:59.750801 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7655-140463795698688"] Jul 10 19:39:59.768795 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:59.768766 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6572p\" (UniqueName: \"kubernetes.io/projected/731b000c-32eb-476b-868b-ce18f5d9c302-kube-api-access-6572p\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:39:59.870226 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:59.870191 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw9z6\" (UniqueName: \"kubernetes.io/projected/20afc1da-f743-4599-838d-1c1c8d0ce141-kube-api-access-tw9z6\") pod \"test-curl-7655-140463795698688\" (UID: \"20afc1da-f743-4599-838d-1c1c8d0ce141\") " pod="openshift-ingress/test-curl-7655-140463795698688" Jul 10 19:39:59.963242 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:59.963164 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="731b000c-32eb-476b-868b-ce18f5d9c302" path="/var/lib/kubelet/pods/731b000c-32eb-476b-868b-ce18f5d9c302/volumes" Jul 10 19:39:59.970660 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:59.970554 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw9z6\" (UniqueName: \"kubernetes.io/projected/20afc1da-f743-4599-838d-1c1c8d0ce141-kube-api-access-tw9z6\") pod \"test-curl-7655-140463795698688\" (UID: \"20afc1da-f743-4599-838d-1c1c8d0ce141\") " pod="openshift-ingress/test-curl-7655-140463795698688" Jul 10 19:39:59.978752 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:39:59.978728 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw9z6\" (UniqueName: \"kubernetes.io/projected/20afc1da-f743-4599-838d-1c1c8d0ce141-kube-api-access-tw9z6\") pod \"test-curl-7655-140463795698688\" (UID: \"20afc1da-f743-4599-838d-1c1c8d0ce141\") " pod="openshift-ingress/test-curl-7655-140463795698688" Jul 10 19:40:00.052026 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:00.051993 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7655-140463795698688" Jul 10 19:40:00.183105 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:00.183082 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7655-140463795698688"] Jul 10 19:40:00.185106 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:40:00.185079 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20afc1da_f743_4599_838d_1c1c8d0ce141.slice/crio-7f9f8bc500493b81cfd979fab82f5af55670b352724861e5996014ab1cf13735 WatchSource:0}: Error finding container 7f9f8bc500493b81cfd979fab82f5af55670b352724861e5996014ab1cf13735: Status 404 returned error can't find the container with id 7f9f8bc500493b81cfd979fab82f5af55670b352724861e5996014ab1cf13735 Jul 10 19:40:00.407889 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:00.407856 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7655-140463795698688" event={"ID":"20afc1da-f743-4599-838d-1c1c8d0ce141","Type":"ContainerStarted","Data":"7f9f8bc500493b81cfd979fab82f5af55670b352724861e5996014ab1cf13735"} Jul 10 19:40:00.408957 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:00.408921 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7655-140463795807024" Jul 10 19:40:00.409069 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:00.408977 2561 scope.go:117] "RemoveContainer" containerID="169c2c468e2d382534a051dddb060b9444cab186f83e5981b7a97dc90ccb5fca" Jul 10 19:40:01.415604 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:01.415570 2561 generic.go:356] "Generic (PLEG): container finished" podID="20afc1da-f743-4599-838d-1c1c8d0ce141" containerID="53c6eeae70623f542bd4504a3c15ecfd50bcd57ca692ea032fd2fe2e8e70bdc6" exitCode=0 Jul 10 19:40:01.416078 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:01.415657 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7655-140463795698688" event={"ID":"20afc1da-f743-4599-838d-1c1c8d0ce141","Type":"ContainerDied","Data":"53c6eeae70623f542bd4504a3c15ecfd50bcd57ca692ea032fd2fe2e8e70bdc6"} Jul 10 19:40:02.200952 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:02.200897 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:40:02.551996 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:02.551967 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7655-140463795698688" Jul 10 19:40:02.575473 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:02.575448 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7655-140463795698688_20afc1da-f743-4599-838d-1c1c8d0ce141/test-curl-7655-140463795698688/0.log" Jul 10 19:40:02.600567 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:02.600527 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7655-140463795698688"] Jul 10 19:40:02.606658 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:02.606626 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7655-140463795698688"] Jul 10 19:40:02.696206 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:02.696171 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw9z6\" (UniqueName: \"kubernetes.io/projected/20afc1da-f743-4599-838d-1c1c8d0ce141-kube-api-access-tw9z6\") pod \"20afc1da-f743-4599-838d-1c1c8d0ce141\" (UID: \"20afc1da-f743-4599-838d-1c1c8d0ce141\") " Jul 10 19:40:02.698434 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:02.698407 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20afc1da-f743-4599-838d-1c1c8d0ce141-kube-api-access-tw9z6" (OuterVolumeSpecName: "kube-api-access-tw9z6") pod "20afc1da-f743-4599-838d-1c1c8d0ce141" (UID: "20afc1da-f743-4599-838d-1c1c8d0ce141"). InnerVolumeSpecName "kube-api-access-tw9z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:40:02.797601 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:02.797566 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw9z6\" (UniqueName: \"kubernetes.io/projected/20afc1da-f743-4599-838d-1c1c8d0ce141-kube-api-access-tw9z6\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:40:03.428062 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:03.428023 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7655-140463795698688" Jul 10 19:40:03.428221 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:03.428074 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f9f8bc500493b81cfd979fab82f5af55670b352724861e5996014ab1cf13735" Jul 10 19:40:03.438643 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:40:03.438617 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7655-140463795698688\" is forbidden: User \"system:node:ip-10-0-130-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-130-74.ec2.internal' and this object" podUID="20afc1da-f743-4599-838d-1c1c8d0ce141" pod="openshift-ingress/test-curl-7655-140463795698688" Jul 10 19:40:03.963191 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:40:03.963154 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7655-140463795698688\" is forbidden: User \"system:node:ip-10-0-130-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-130-74.ec2.internal' and this object" podUID="20afc1da-f743-4599-838d-1c1c8d0ce141" pod="openshift-ingress/test-curl-7655-140463795698688" Jul 10 19:40:03.964225 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:03.964198 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20afc1da-f743-4599-838d-1c1c8d0ce141" path="/var/lib/kubelet/pods/20afc1da-f743-4599-838d-1c1c8d0ce141/volumes" Jul 10 19:40:05.072483 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:05.072452 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7655-140463795699552"] Jul 10 19:40:05.072881 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:05.072763 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20afc1da-f743-4599-838d-1c1c8d0ce141" containerName="test-curl-7655-140463795698688" Jul 10 19:40:05.072881 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:05.072773 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="20afc1da-f743-4599-838d-1c1c8d0ce141" containerName="test-curl-7655-140463795698688" Jul 10 19:40:05.072881 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:05.072828 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="20afc1da-f743-4599-838d-1c1c8d0ce141" containerName="test-curl-7655-140463795698688" Jul 10 19:40:05.075774 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:05.075758 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7655-140463795699552" Jul 10 19:40:05.079695 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:05.079673 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-q7hmp\"" Jul 10 19:40:05.083897 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:05.083872 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7655-140463795699552"] Jul 10 19:40:05.220317 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:05.220281 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68qcs\" (UniqueName: \"kubernetes.io/projected/c777d3c4-153e-4af7-b853-0a433cc7c927-kube-api-access-68qcs\") pod \"test-curl-7655-140463795699552\" (UID: \"c777d3c4-153e-4af7-b853-0a433cc7c927\") " pod="openshift-ingress/test-curl-7655-140463795699552" Jul 10 19:40:05.321025 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:05.320975 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68qcs\" (UniqueName: \"kubernetes.io/projected/c777d3c4-153e-4af7-b853-0a433cc7c927-kube-api-access-68qcs\") pod \"test-curl-7655-140463795699552\" (UID: \"c777d3c4-153e-4af7-b853-0a433cc7c927\") " pod="openshift-ingress/test-curl-7655-140463795699552" Jul 10 19:40:05.330301 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:05.330233 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68qcs\" (UniqueName: \"kubernetes.io/projected/c777d3c4-153e-4af7-b853-0a433cc7c927-kube-api-access-68qcs\") pod \"test-curl-7655-140463795699552\" (UID: \"c777d3c4-153e-4af7-b853-0a433cc7c927\") " pod="openshift-ingress/test-curl-7655-140463795699552" Jul 10 19:40:05.385665 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:05.385632 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7655-140463795699552" Jul 10 19:40:05.519700 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:05.519670 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7655-140463795699552"] Jul 10 19:40:05.521474 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:40:05.521447 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc777d3c4_153e_4af7_b853_0a433cc7c927.slice/crio-e6d2b424323b5925a7a4890dd5b082cbe286e2f0dc2908a6eeb467bb4d80520e WatchSource:0}: Error finding container e6d2b424323b5925a7a4890dd5b082cbe286e2f0dc2908a6eeb467bb4d80520e: Status 404 returned error can't find the container with id e6d2b424323b5925a7a4890dd5b082cbe286e2f0dc2908a6eeb467bb4d80520e Jul 10 19:40:05.754038 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:05.753978 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" podUID="a2faff33-7cc3-4ab5-85c1-238a39dc0861" containerName="main" containerID="cri-o://7a883f45364cbe599614cd58c58f23b24f8d9804b429a09e3f9a503588caf1d6" gracePeriod=45 Jul 10 19:40:06.005347 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.005288 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:40:06.128023 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.127983 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmtr4\" (UniqueName: \"kubernetes.io/projected/a2faff33-7cc3-4ab5-85c1-238a39dc0861-kube-api-access-kmtr4\") pod \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " Jul 10 19:40:06.128483 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.128035 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2faff33-7cc3-4ab5-85c1-238a39dc0861-tls-certs\") pod \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " Jul 10 19:40:06.128483 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.128057 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-dshm\") pod \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " Jul 10 19:40:06.128483 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.128119 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-tmp-dir\") pod \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " Jul 10 19:40:06.128483 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.128148 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-home\") pod \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " Jul 10 19:40:06.128483 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.128183 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-model-cache\") pod \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\" (UID: \"a2faff33-7cc3-4ab5-85c1-238a39dc0861\") " Jul 10 19:40:06.128483 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.128396 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a2faff33-7cc3-4ab5-85c1-238a39dc0861" (UID: "a2faff33-7cc3-4ab5-85c1-238a39dc0861"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:06.128483 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.128425 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-home" (OuterVolumeSpecName: "home") pod "a2faff33-7cc3-4ab5-85c1-238a39dc0861" (UID: "a2faff33-7cc3-4ab5-85c1-238a39dc0861"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:06.128782 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.128526 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-model-cache" (OuterVolumeSpecName: "model-cache") pod "a2faff33-7cc3-4ab5-85c1-238a39dc0861" (UID: "a2faff33-7cc3-4ab5-85c1-238a39dc0861"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:06.130343 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.130311 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-dshm" (OuterVolumeSpecName: "dshm") pod "a2faff33-7cc3-4ab5-85c1-238a39dc0861" (UID: "a2faff33-7cc3-4ab5-85c1-238a39dc0861"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:06.130343 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.130314 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2faff33-7cc3-4ab5-85c1-238a39dc0861-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a2faff33-7cc3-4ab5-85c1-238a39dc0861" (UID: "a2faff33-7cc3-4ab5-85c1-238a39dc0861"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:40:06.130470 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.130346 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2faff33-7cc3-4ab5-85c1-238a39dc0861-kube-api-access-kmtr4" (OuterVolumeSpecName: "kube-api-access-kmtr4") pod "a2faff33-7cc3-4ab5-85c1-238a39dc0861" (UID: "a2faff33-7cc3-4ab5-85c1-238a39dc0861"). InnerVolumeSpecName "kube-api-access-kmtr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:40:06.229075 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.229033 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-tmp-dir\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:40:06.229075 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.229067 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-home\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:40:06.229075 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.229077 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-model-cache\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:40:06.229075 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.229087 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmtr4\" (UniqueName: \"kubernetes.io/projected/a2faff33-7cc3-4ab5-85c1-238a39dc0861-kube-api-access-kmtr4\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:40:06.229387 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.229095 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a2faff33-7cc3-4ab5-85c1-238a39dc0861-tls-certs\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:40:06.229387 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.229103 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a2faff33-7cc3-4ab5-85c1-238a39dc0861-dshm\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:40:06.452232 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.452192 2561 generic.go:356] "Generic (PLEG): container finished" podID="c777d3c4-153e-4af7-b853-0a433cc7c927" containerID="43bb203a693c41e1887456a368fcb3dce75faf2c9d5dca3c3b164d026a1c397b" exitCode=0 Jul 10 19:40:06.452429 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.452277 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7655-140463795699552" event={"ID":"c777d3c4-153e-4af7-b853-0a433cc7c927","Type":"ContainerDied","Data":"43bb203a693c41e1887456a368fcb3dce75faf2c9d5dca3c3b164d026a1c397b"} Jul 10 19:40:06.452429 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.452316 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7655-140463795699552" event={"ID":"c777d3c4-153e-4af7-b853-0a433cc7c927","Type":"ContainerStarted","Data":"e6d2b424323b5925a7a4890dd5b082cbe286e2f0dc2908a6eeb467bb4d80520e"} Jul 10 19:40:06.453654 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.453630 2561 generic.go:356] "Generic (PLEG): container finished" podID="a2faff33-7cc3-4ab5-85c1-238a39dc0861" containerID="7a883f45364cbe599614cd58c58f23b24f8d9804b429a09e3f9a503588caf1d6" exitCode=0 Jul 10 19:40:06.453758 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.453667 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" event={"ID":"a2faff33-7cc3-4ab5-85c1-238a39dc0861","Type":"ContainerDied","Data":"7a883f45364cbe599614cd58c58f23b24f8d9804b429a09e3f9a503588caf1d6"} Jul 10 19:40:06.453758 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.453682 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" Jul 10 19:40:06.453758 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.453695 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld" event={"ID":"a2faff33-7cc3-4ab5-85c1-238a39dc0861","Type":"ContainerDied","Data":"446cdd22940697c73c8868ec87e8438db704a921311baf291a46ef45018d8147"} Jul 10 19:40:06.453758 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.453716 2561 scope.go:117] "RemoveContainer" containerID="7a883f45364cbe599614cd58c58f23b24f8d9804b429a09e3f9a503588caf1d6" Jul 10 19:40:06.463083 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.463063 2561 scope.go:117] "RemoveContainer" containerID="7a883f45364cbe599614cd58c58f23b24f8d9804b429a09e3f9a503588caf1d6" Jul 10 19:40:06.463333 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:40:06.463314 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a883f45364cbe599614cd58c58f23b24f8d9804b429a09e3f9a503588caf1d6\": container with ID starting with 7a883f45364cbe599614cd58c58f23b24f8d9804b429a09e3f9a503588caf1d6 not found: ID does not exist" containerID="7a883f45364cbe599614cd58c58f23b24f8d9804b429a09e3f9a503588caf1d6" Jul 10 19:40:06.463397 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.463345 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a883f45364cbe599614cd58c58f23b24f8d9804b429a09e3f9a503588caf1d6"} err="failed to get container status \"7a883f45364cbe599614cd58c58f23b24f8d9804b429a09e3f9a503588caf1d6\": rpc error: code = NotFound desc = could not find container \"7a883f45364cbe599614cd58c58f23b24f8d9804b429a09e3f9a503588caf1d6\": container with ID starting with 7a883f45364cbe599614cd58c58f23b24f8d9804b429a09e3f9a503588caf1d6 not found: ID does not exist" Jul 10 19:40:06.482577 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.482552 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld"] Jul 10 19:40:06.486611 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.486590 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-b1f450/rate-test-model-b1f450-kserve-644487f7f5-79zld"] Jul 10 19:40:06.618368 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.618275 2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" podUID="9b75041e-6e3a-49dd-81e6-9232008743e8" containerName="main" containerID="cri-o://c49fb1789fc9459cf154fa200eeb6517159ef9827ef80d0fe2ec3ffb3a022893" gracePeriod=45 Jul 10 19:40:06.874111 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:06.874040 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:40:07.036848 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.036814 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-dshm\") pod \"9b75041e-6e3a-49dd-81e6-9232008743e8\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " Jul 10 19:40:07.037066 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.036890 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b75041e-6e3a-49dd-81e6-9232008743e8-tls-certs\") pod \"9b75041e-6e3a-49dd-81e6-9232008743e8\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " Jul 10 19:40:07.037066 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.036926 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzw9x\" (UniqueName: \"kubernetes.io/projected/9b75041e-6e3a-49dd-81e6-9232008743e8-kube-api-access-jzw9x\") pod \"9b75041e-6e3a-49dd-81e6-9232008743e8\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " Jul 10 19:40:07.037066 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.036979 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-model-cache\") pod \"9b75041e-6e3a-49dd-81e6-9232008743e8\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " Jul 10 19:40:07.037066 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.037025 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-tmp-dir\") pod \"9b75041e-6e3a-49dd-81e6-9232008743e8\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " Jul 10 19:40:07.037066 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.037051 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-home\") pod \"9b75041e-6e3a-49dd-81e6-9232008743e8\" (UID: \"9b75041e-6e3a-49dd-81e6-9232008743e8\") " Jul 10 19:40:07.037309 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.037274 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b75041e-6e3a-49dd-81e6-9232008743e8" (UID: "9b75041e-6e3a-49dd-81e6-9232008743e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:07.037376 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.037303 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b75041e-6e3a-49dd-81e6-9232008743e8" (UID: "9b75041e-6e3a-49dd-81e6-9232008743e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:07.037436 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.037377 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-home" (OuterVolumeSpecName: "home") pod "9b75041e-6e3a-49dd-81e6-9232008743e8" (UID: "9b75041e-6e3a-49dd-81e6-9232008743e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:07.039269 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.039248 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-dshm" (OuterVolumeSpecName: "dshm") pod "9b75041e-6e3a-49dd-81e6-9232008743e8" (UID: "9b75041e-6e3a-49dd-81e6-9232008743e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 19:40:07.039365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.039305 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b75041e-6e3a-49dd-81e6-9232008743e8-kube-api-access-jzw9x" (OuterVolumeSpecName: "kube-api-access-jzw9x") pod "9b75041e-6e3a-49dd-81e6-9232008743e8" (UID: "9b75041e-6e3a-49dd-81e6-9232008743e8"). InnerVolumeSpecName "kube-api-access-jzw9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:40:07.039365 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.039310 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b75041e-6e3a-49dd-81e6-9232008743e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b75041e-6e3a-49dd-81e6-9232008743e8" (UID: "9b75041e-6e3a-49dd-81e6-9232008743e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 19:40:07.138756 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.138657 2561 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b75041e-6e3a-49dd-81e6-9232008743e8-tls-certs\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:40:07.138756 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.138691 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzw9x\" (UniqueName: \"kubernetes.io/projected/9b75041e-6e3a-49dd-81e6-9232008743e8-kube-api-access-jzw9x\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:40:07.138756 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.138705 2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-model-cache\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:40:07.138756 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.138732 2561 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-tmp-dir\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:40:07.138756 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.138740 2561 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-home\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:40:07.138756 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.138747 2561 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b75041e-6e3a-49dd-81e6-9232008743e8-dshm\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:40:07.459609 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.459517 2561 generic.go:356] "Generic (PLEG): container finished" podID="9b75041e-6e3a-49dd-81e6-9232008743e8" containerID="c49fb1789fc9459cf154fa200eeb6517159ef9827ef80d0fe2ec3ffb3a022893" exitCode=0 Jul 10 19:40:07.459609 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.459563 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" Jul 10 19:40:07.459823 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.459603 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" event={"ID":"9b75041e-6e3a-49dd-81e6-9232008743e8","Type":"ContainerDied","Data":"c49fb1789fc9459cf154fa200eeb6517159ef9827ef80d0fe2ec3ffb3a022893"} Jul 10 19:40:07.459823 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.459644 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq" event={"ID":"9b75041e-6e3a-49dd-81e6-9232008743e8","Type":"ContainerDied","Data":"59e48857c40b1de867cbe22822041cecbe61e6e8843bdd848dbd76316cde41f0"} Jul 10 19:40:07.459823 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.459661 2561 scope.go:117] "RemoveContainer" containerID="c49fb1789fc9459cf154fa200eeb6517159ef9827ef80d0fe2ec3ffb3a022893" Jul 10 19:40:07.469696 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.469665 2561 scope.go:117] "RemoveContainer" containerID="c49fb1789fc9459cf154fa200eeb6517159ef9827ef80d0fe2ec3ffb3a022893" Jul 10 19:40:07.470007 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:40:07.469986 2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49fb1789fc9459cf154fa200eeb6517159ef9827ef80d0fe2ec3ffb3a022893\": container with ID starting with c49fb1789fc9459cf154fa200eeb6517159ef9827ef80d0fe2ec3ffb3a022893 not found: ID does not exist" containerID="c49fb1789fc9459cf154fa200eeb6517159ef9827ef80d0fe2ec3ffb3a022893" Jul 10 19:40:07.470076 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.470019 2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49fb1789fc9459cf154fa200eeb6517159ef9827ef80d0fe2ec3ffb3a022893"} err="failed to get container status \"c49fb1789fc9459cf154fa200eeb6517159ef9827ef80d0fe2ec3ffb3a022893\": rpc error: code = NotFound desc = could not find container \"c49fb1789fc9459cf154fa200eeb6517159ef9827ef80d0fe2ec3ffb3a022893\": container with ID starting with c49fb1789fc9459cf154fa200eeb6517159ef9827ef80d0fe2ec3ffb3a022893 not found: ID does not exist" Jul 10 19:40:07.484691 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.484658 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq"] Jul 10 19:40:07.490596 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.490567 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b02417/rate-test-model-b02417-kserve-54c6bf7ff5-mjdhq"] Jul 10 19:40:07.598823 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.598797 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7655-140463795699552" Jul 10 19:40:07.620671 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.620627 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7655-140463795699552_c777d3c4-153e-4af7-b853-0a433cc7c927/test-curl-7655-140463795699552/0.log" Jul 10 19:40:07.645551 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.645513 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7655-140463795699552"] Jul 10 19:40:07.651435 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.651404 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7655-140463795699552"] Jul 10 19:40:07.681217 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.681158 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:07.745220 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.745190 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68qcs\" (UniqueName: \"kubernetes.io/projected/c777d3c4-153e-4af7-b853-0a433cc7c927-kube-api-access-68qcs\") pod \"c777d3c4-153e-4af7-b853-0a433cc7c927\" (UID: \"c777d3c4-153e-4af7-b853-0a433cc7c927\") " Jul 10 19:40:07.747397 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.747377 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c777d3c4-153e-4af7-b853-0a433cc7c927-kube-api-access-68qcs" (OuterVolumeSpecName: "kube-api-access-68qcs") pod "c777d3c4-153e-4af7-b853-0a433cc7c927" (UID: "c777d3c4-153e-4af7-b853-0a433cc7c927"). InnerVolumeSpecName "kube-api-access-68qcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:40:07.766413 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.766383 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7655-140463795699552"] Jul 10 19:40:07.766717 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.766705 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b75041e-6e3a-49dd-81e6-9232008743e8" containerName="main" Jul 10 19:40:07.766761 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.766719 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b75041e-6e3a-49dd-81e6-9232008743e8" containerName="main" Jul 10 19:40:07.766761 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.766729 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2faff33-7cc3-4ab5-85c1-238a39dc0861" containerName="main" Jul 10 19:40:07.766761 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.766735 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2faff33-7cc3-4ab5-85c1-238a39dc0861" containerName="main" Jul 10 19:40:07.766761 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.766744 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c777d3c4-153e-4af7-b853-0a433cc7c927" containerName="test-curl-7655-140463795699552" Jul 10 19:40:07.766761 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.766749 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c777d3c4-153e-4af7-b853-0a433cc7c927" containerName="test-curl-7655-140463795699552" Jul 10 19:40:07.766917 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.766799 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c777d3c4-153e-4af7-b853-0a433cc7c927" containerName="test-curl-7655-140463795699552" Jul 10 19:40:07.766917 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.766808 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2faff33-7cc3-4ab5-85c1-238a39dc0861" containerName="main" Jul 10 19:40:07.766917 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.766815 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b75041e-6e3a-49dd-81e6-9232008743e8" containerName="main" Jul 10 19:40:07.769860 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.769844 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7655-140463795699552" Jul 10 19:40:07.775948 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.775902 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7655-140463795699552"] Jul 10 19:40:07.846211 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.846170 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68qcs\" (UniqueName: \"kubernetes.io/projected/c777d3c4-153e-4af7-b853-0a433cc7c927-kube-api-access-68qcs\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:40:07.946651 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.946614 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ncnl\" (UniqueName: \"kubernetes.io/projected/38abd521-9426-4c21-8b5d-e4bfd027f025-kube-api-access-5ncnl\") pod \"test-curl-7655-140463795699552\" (UID: \"38abd521-9426-4c21-8b5d-e4bfd027f025\") " pod="openshift-ingress/test-curl-7655-140463795699552" Jul 10 19:40:07.947655 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.947618 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:07.962979 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.962948 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b75041e-6e3a-49dd-81e6-9232008743e8" path="/var/lib/kubelet/pods/9b75041e-6e3a-49dd-81e6-9232008743e8/volumes" Jul 10 19:40:07.963406 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.963386 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2faff33-7cc3-4ab5-85c1-238a39dc0861" path="/var/lib/kubelet/pods/a2faff33-7cc3-4ab5-85c1-238a39dc0861/volumes" Jul 10 19:40:07.963729 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:07.963712 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c777d3c4-153e-4af7-b853-0a433cc7c927" path="/var/lib/kubelet/pods/c777d3c4-153e-4af7-b853-0a433cc7c927/volumes" Jul 10 19:40:08.047209 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:08.047107 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ncnl\" (UniqueName: \"kubernetes.io/projected/38abd521-9426-4c21-8b5d-e4bfd027f025-kube-api-access-5ncnl\") pod \"test-curl-7655-140463795699552\" (UID: \"38abd521-9426-4c21-8b5d-e4bfd027f025\") " pod="openshift-ingress/test-curl-7655-140463795699552" Jul 10 19:40:08.055972 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:08.055923 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ncnl\" (UniqueName: \"kubernetes.io/projected/38abd521-9426-4c21-8b5d-e4bfd027f025-kube-api-access-5ncnl\") pod \"test-curl-7655-140463795699552\" (UID: \"38abd521-9426-4c21-8b5d-e4bfd027f025\") " pod="openshift-ingress/test-curl-7655-140463795699552" Jul 10 19:40:08.079360 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:08.079331 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7655-140463795699552" Jul 10 19:40:08.231717 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:08.231689 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7655-140463795699552"] Jul 10 19:40:08.233781 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:40:08.233755 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38abd521_9426_4c21_8b5d_e4bfd027f025.slice/crio-f417fb232cbbc6ab07a5ae2fcf74d2842d358a304fb36963a711ab78e5bcd7aa WatchSource:0}: Error finding container f417fb232cbbc6ab07a5ae2fcf74d2842d358a304fb36963a711ab78e5bcd7aa: Status 404 returned error can't find the container with id f417fb232cbbc6ab07a5ae2fcf74d2842d358a304fb36963a711ab78e5bcd7aa Jul 10 19:40:08.466520 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:08.466434 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7655-140463795699552" event={"ID":"38abd521-9426-4c21-8b5d-e4bfd027f025","Type":"ContainerStarted","Data":"f417fb232cbbc6ab07a5ae2fcf74d2842d358a304fb36963a711ab78e5bcd7aa"} Jul 10 19:40:08.467717 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:08.467691 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7655-140463795699552" Jul 10 19:40:08.467839 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:08.467730 2561 scope.go:117] "RemoveContainer" containerID="43bb203a693c41e1887456a368fcb3dce75faf2c9d5dca3c3b164d026a1c397b" Jul 10 19:40:09.475531 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:09.475494 2561 generic.go:356] "Generic (PLEG): container finished" podID="38abd521-9426-4c21-8b5d-e4bfd027f025" containerID="91a81154ad1d9c1f87b9bc17894b3944549fee20afb839e2195d9e553c1f96ed" exitCode=0 Jul 10 19:40:09.475978 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:09.475580 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7655-140463795699552" event={"ID":"38abd521-9426-4c21-8b5d-e4bfd027f025","Type":"ContainerDied","Data":"91a81154ad1d9c1f87b9bc17894b3944549fee20afb839e2195d9e553c1f96ed"} Jul 10 19:40:10.619296 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:10.619271 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7655-140463795699552" Jul 10 19:40:10.642648 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:10.642621 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7655-140463795699552_38abd521-9426-4c21-8b5d-e4bfd027f025/test-curl-7655-140463795699552/0.log" Jul 10 19:40:10.665464 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:10.665435 2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7655-140463795699552"] Jul 10 19:40:10.668949 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:10.668908 2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ncnl\" (UniqueName: \"kubernetes.io/projected/38abd521-9426-4c21-8b5d-e4bfd027f025-kube-api-access-5ncnl\") pod \"38abd521-9426-4c21-8b5d-e4bfd027f025\" (UID: \"38abd521-9426-4c21-8b5d-e4bfd027f025\") " Jul 10 19:40:10.669813 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:10.669787 2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7655-140463795699552"] Jul 10 19:40:10.671110 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:10.671085 2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38abd521-9426-4c21-8b5d-e4bfd027f025-kube-api-access-5ncnl" (OuterVolumeSpecName: "kube-api-access-5ncnl") pod "38abd521-9426-4c21-8b5d-e4bfd027f025" (UID: "38abd521-9426-4c21-8b5d-e4bfd027f025"). InnerVolumeSpecName "kube-api-access-5ncnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 19:40:10.769488 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:10.769447 2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ncnl\" (UniqueName: \"kubernetes.io/projected/38abd521-9426-4c21-8b5d-e4bfd027f025-kube-api-access-5ncnl\") on node \"ip-10-0-130-74.ec2.internal\" DevicePath \"\"" Jul 10 19:40:10.919994 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:10.919927 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:11.338857 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:11.338822 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:11.493539 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:11.493512 2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7655-140463795699552" Jul 10 19:40:11.493711 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:11.493523 2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f417fb232cbbc6ab07a5ae2fcf74d2842d358a304fb36963a711ab78e5bcd7aa" Jul 10 19:40:11.503484 ip-10-0-130-74 kubenswrapper[2561]: E0710 19:40:11.503422 2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7655-140463795699552\" is forbidden: User \"system:node:ip-10-0-130-74.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-130-74.ec2.internal' and this object" podUID="38abd521-9426-4c21-8b5d-e4bfd027f025" pod="openshift-ingress/test-curl-7655-140463795699552" Jul 10 19:40:11.503707 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:11.503688 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:11.604375 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:11.604288 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:11.768451 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:11.768408 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:11.963295 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:11.963214 2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38abd521-9426-4c21-8b5d-e4bfd027f025" path="/var/lib/kubelet/pods/38abd521-9426-4c21-8b5d-e4bfd027f025/volumes" Jul 10 19:40:15.189670 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:15.189634 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:40:17.705807 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:17.705777 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:17.803298 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:17.803253 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:17.979305 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:17.979217 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:18.193708 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:18.193669 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:18.361174 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:18.361140 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:18.740490 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:18.740399 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:18.840782 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:18.840743 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:18.957579 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:18.957550 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:19.140505 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:19.140469 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:25.351141 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:25.351105 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:25.450918 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:25.450868 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:25.630383 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:25.630296 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:25.828490 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:25.828450 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:26.255428 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:26.255382 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bbc49449f-snpzm_44e764e6-1713-43ea-b2c6-a90c293a8d66/authorino/0.log" Jul 10 19:40:28.194448 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:28.194402 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-4kfc2"] Jul 10 19:40:31.854405 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:31.854374 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7668cd484b-9qvkc_22acb39a-d928-4715-a844-945c72839563/copy-manifests/0.log" Jul 10 19:40:31.878423 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:31.878400 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7668cd484b-9qvkc_22acb39a-d928-4715-a844-945c72839563/manager/0.log" Jul 10 19:40:33.215046 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:33.215010 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd_8631bef5-d160-4ae4-88b0-b137c25ced1e/util/0.log" Jul 10 19:40:33.223240 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:33.223213 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd_8631bef5-d160-4ae4-88b0-b137c25ced1e/pull/0.log" Jul 10 19:40:33.230707 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:33.230685 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd_8631bef5-d160-4ae4-88b0-b137c25ced1e/extract/0.log" Jul 10 19:40:33.343787 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:33.343762 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs_855f697c-e062-4cac-88e2-9f11d5786c68/util/0.log" Jul 10 19:40:33.352183 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:33.352163 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs_855f697c-e062-4cac-88e2-9f11d5786c68/pull/0.log" Jul 10 19:40:33.359586 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:33.359559 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs_855f697c-e062-4cac-88e2-9f11d5786c68/extract/0.log" Jul 10 19:40:33.469812 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:33.469735 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x_1fb12101-ccd5-474e-b4b7-b05700be5463/util/0.log" Jul 10 19:40:33.476498 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:33.476467 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x_1fb12101-ccd5-474e-b4b7-b05700be5463/pull/0.log" Jul 10 19:40:33.484207 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:33.484181 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x_1fb12101-ccd5-474e-b4b7-b05700be5463/extract/0.log" Jul 10 19:40:33.600686 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:33.600649 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9_a5081e81-09af-4b03-a57e-1f1e89455699/util/0.log" Jul 10 19:40:33.607603 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:33.607583 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9_a5081e81-09af-4b03-a57e-1f1e89455699/pull/0.log" Jul 10 19:40:33.613901 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:33.613876 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9_a5081e81-09af-4b03-a57e-1f1e89455699/extract/0.log" Jul 10 19:40:33.738712 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:33.738608 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bbc49449f-snpzm_44e764e6-1713-43ea-b2c6-a90c293a8d66/authorino/0.log" Jul 10 19:40:34.072306 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:34.072281 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:40:34.077544 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:34.077522 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-knrq9_7c3b76b0-8cb8-489c-ae2e-a8e78982c24b/ovn-acl-logging/0.log" Jul 10 19:40:34.096650 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:34.096624 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-8wrx4_e249aad8-34ff-4221-876e-2d907faa19f3/kuadrant-console-plugin/0.log" Jul 10 19:40:34.463575 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:34.463493 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-4kfc2_45700ba9-9c1d-4475-9267-6a89879dc36e/limitador/0.log" Jul 10 19:40:35.153664 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:35.153637 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b4fc6b9d-7s88n_36c31443-af47-4c59-a1d4-a00aa0f29dee/kube-auth-proxy/0.log" Jul 10 19:40:36.624591 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:36.624556 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-xpcvm_7cec515a-619d-4d1a-b7d8-8505938a988f/main/0.log" Jul 10 19:40:40.151022 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.150982 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z44r6/must-gather-9d8h4"] Jul 10 19:40:40.151399 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.151283 2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38abd521-9426-4c21-8b5d-e4bfd027f025" containerName="test-curl-7655-140463795699552" Jul 10 19:40:40.151399 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.151294 2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="38abd521-9426-4c21-8b5d-e4bfd027f025" containerName="test-curl-7655-140463795699552" Jul 10 19:40:40.151399 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.151353 2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="38abd521-9426-4c21-8b5d-e4bfd027f025" containerName="test-curl-7655-140463795699552" Jul 10 19:40:40.154294 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.154266 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z44r6/must-gather-9d8h4" Jul 10 19:40:40.157223 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.157198 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z44r6\"/\"kube-root-ca.crt\"" Jul 10 19:40:40.157351 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.157244 2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z44r6\"/\"default-dockercfg-q7qdh\"" Jul 10 19:40:40.157351 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.157244 2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z44r6\"/\"openshift-service-ca.crt\"" Jul 10 19:40:40.166583 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.166557 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z44r6/must-gather-9d8h4"] Jul 10 19:40:40.173742 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.173714 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:40.223998 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.223967 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/06180cc4-445a-4802-8bc1-ac1603163131-must-gather-output\") pod \"must-gather-9d8h4\" (UID: \"06180cc4-445a-4802-8bc1-ac1603163131\") " pod="openshift-must-gather-z44r6/must-gather-9d8h4" Jul 10 19:40:40.224167 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.224029 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czqq9\" (UniqueName: \"kubernetes.io/projected/06180cc4-445a-4802-8bc1-ac1603163131-kube-api-access-czqq9\") pod \"must-gather-9d8h4\" (UID: \"06180cc4-445a-4802-8bc1-ac1603163131\") " pod="openshift-must-gather-z44r6/must-gather-9d8h4" Jul 10 19:40:40.231667 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.231635 2561 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" limit=1024 Jul 10 19:40:40.324736 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.324705 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czqq9\" (UniqueName: \"kubernetes.io/projected/06180cc4-445a-4802-8bc1-ac1603163131-kube-api-access-czqq9\") pod \"must-gather-9d8h4\" (UID: \"06180cc4-445a-4802-8bc1-ac1603163131\") " pod="openshift-must-gather-z44r6/must-gather-9d8h4" Jul 10 19:40:40.324919 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.324770 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/06180cc4-445a-4802-8bc1-ac1603163131-must-gather-output\") pod \"must-gather-9d8h4\" (UID: \"06180cc4-445a-4802-8bc1-ac1603163131\") " pod="openshift-must-gather-z44r6/must-gather-9d8h4" Jul 10 19:40:40.325104 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.325083 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/06180cc4-445a-4802-8bc1-ac1603163131-must-gather-output\") pod \"must-gather-9d8h4\" (UID: \"06180cc4-445a-4802-8bc1-ac1603163131\") " pod="openshift-must-gather-z44r6/must-gather-9d8h4" Jul 10 19:40:40.333678 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.333656 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czqq9\" (UniqueName: \"kubernetes.io/projected/06180cc4-445a-4802-8bc1-ac1603163131-kube-api-access-czqq9\") pod \"must-gather-9d8h4\" (UID: \"06180cc4-445a-4802-8bc1-ac1603163131\") " pod="openshift-must-gather-z44r6/must-gather-9d8h4" Jul 10 19:40:40.463704 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.463607 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z44r6/must-gather-9d8h4" Jul 10 19:40:40.599189 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.599156 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z44r6/must-gather-9d8h4"] Jul 10 19:40:40.636083 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:40.636050 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z44r6/must-gather-9d8h4" event={"ID":"06180cc4-445a-4802-8bc1-ac1603163131","Type":"ContainerStarted","Data":"0b01f43a318913b5de40fd2193a66c6111b0b9914e45b3afc52c0546d3dc79ca"} Jul 10 19:40:41.644455 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:41.644405 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z44r6/must-gather-9d8h4" event={"ID":"06180cc4-445a-4802-8bc1-ac1603163131","Type":"ContainerStarted","Data":"4bede7dc443a3d3b8a45d81fd6618a23aa948327c1ff3aed9ac2ba80900b0c57"} Jul 10 19:40:42.652262 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:42.652228 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z44r6/must-gather-9d8h4" event={"ID":"06180cc4-445a-4802-8bc1-ac1603163131","Type":"ContainerStarted","Data":"f173d4c7300e57678d0d08e304a39f37382c3c57f66b8e822077a141fa552dfa"} Jul 10 19:40:42.670080 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:42.670022 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-z44r6/must-gather-9d8h4" podStartSLOduration=1.7559174720000001 podStartE2EDuration="2.669999299s" totalImagesPullingTime="914.081827ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 19:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:40:42.668864771 +0000 UTC m=+2409.270006907" watchObservedRunningTime="2026-07-10 19:40:42.669999299 +0000 UTC m=+2409.271141435" Jul 10 19:40:43.225883 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:43.225847 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j62sj_2509af5e-c021-4342-b2f3-7448be0ce972/global-pull-secret-syncer/0.log" Jul 10 19:40:43.342539 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:43.342505 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bxw74_f9ffcad3-0a20-40c1-9383-454676ea46aa/konnectivity-agent/0.log" Jul 10 19:40:43.444083 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:43.444050 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-74.ec2.internal_7651e5e5e93604167127627171e7b52f/haproxy/0.log" Jul 10 19:40:48.206651 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:48.206620 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd_8631bef5-d160-4ae4-88b0-b137c25ced1e/extract/0.log" Jul 10 19:40:48.237088 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:48.237057 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd_8631bef5-d160-4ae4-88b0-b137c25ced1e/util/0.log" Jul 10 19:40:48.267059 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:48.267016 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b75928hwd_8631bef5-d160-4ae4-88b0-b137c25ced1e/pull/0.log" Jul 10 19:40:48.313805 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:48.313780 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs_855f697c-e062-4cac-88e2-9f11d5786c68/extract/0.log" Jul 10 19:40:48.340863 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:48.340786 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs_855f697c-e062-4cac-88e2-9f11d5786c68/util/0.log" Jul 10 19:40:48.368958 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:48.368913 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e02mcjs_855f697c-e062-4cac-88e2-9f11d5786c68/pull/0.log" Jul 10 19:40:48.404403 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:48.404368 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x_1fb12101-ccd5-474e-b4b7-b05700be5463/extract/0.log" Jul 10 19:40:48.438239 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:48.438157 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x_1fb12101-ccd5-474e-b4b7-b05700be5463/util/0.log" Jul 10 19:40:48.469530 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:48.469339 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73p8n7x_1fb12101-ccd5-474e-b4b7-b05700be5463/pull/0.log" Jul 10 19:40:48.510069 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:48.509990 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9_a5081e81-09af-4b03-a57e-1f1e89455699/extract/0.log" Jul 10 19:40:48.543841 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:48.543806 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9_a5081e81-09af-4b03-a57e-1f1e89455699/util/0.log" Jul 10 19:40:48.587093 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:48.587058 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19btr9_a5081e81-09af-4b03-a57e-1f1e89455699/pull/0.log" Jul 10 19:40:49.112857 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:49.112811 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-bbc49449f-snpzm_44e764e6-1713-43ea-b2c6-a90c293a8d66/authorino/0.log" Jul 10 19:40:49.231772 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:49.231742 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-8wrx4_e249aad8-34ff-4221-876e-2d907faa19f3/kuadrant-console-plugin/0.log" Jul 10 19:40:49.435591 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:49.435499 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-4kfc2_45700ba9-9c1d-4475-9267-6a89879dc36e/limitador/0.log" Jul 10 19:40:51.724702 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:51.724672 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xs249_42e4fcd1-edab-49e1-9ac4-8462e1a43d2f/node-exporter/0.log" Jul 10 19:40:51.753758 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:51.753721 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xs249_42e4fcd1-edab-49e1-9ac4-8462e1a43d2f/kube-rbac-proxy/0.log" Jul 10 19:40:51.778076 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:51.778050 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xs249_42e4fcd1-edab-49e1-9ac4-8462e1a43d2f/init-textfile/0.log" Jul 10 19:40:52.279797 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:52.279767 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57769bbb47-876fw_6f3f76d4-602c-4c81-9a2a-c8679e91ac00/thanos-query/0.log" Jul 10 19:40:52.306003 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:52.305968 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57769bbb47-876fw_6f3f76d4-602c-4c81-9a2a-c8679e91ac00/kube-rbac-proxy-web/0.log" Jul 10 19:40:52.333897 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:52.333839 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57769bbb47-876fw_6f3f76d4-602c-4c81-9a2a-c8679e91ac00/kube-rbac-proxy/0.log" Jul 10 19:40:52.360429 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:52.360404 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57769bbb47-876fw_6f3f76d4-602c-4c81-9a2a-c8679e91ac00/prom-label-proxy/0.log" Jul 10 19:40:52.386869 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:52.386837 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57769bbb47-876fw_6f3f76d4-602c-4c81-9a2a-c8679e91ac00/kube-rbac-proxy-rules/0.log" Jul 10 19:40:52.416185 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:52.416154 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57769bbb47-876fw_6f3f76d4-602c-4c81-9a2a-c8679e91ac00/kube-rbac-proxy-metrics/0.log" Jul 10 19:40:53.419054 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.419019 2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s"] Jul 10 19:40:53.428996 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.428964 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" Jul 10 19:40:53.437438 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.437351 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s"] Jul 10 19:40:53.464885 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.464839 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmqtx\" (UniqueName: \"kubernetes.io/projected/1a12882f-bc9e-40d3-a9f2-de5778150aaf-kube-api-access-fmqtx\") pod \"perf-node-gather-daemonset-jtl7s\" (UID: \"1a12882f-bc9e-40d3-a9f2-de5778150aaf\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" Jul 10 19:40:53.465110 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.464921 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a12882f-bc9e-40d3-a9f2-de5778150aaf-sys\") pod \"perf-node-gather-daemonset-jtl7s\" (UID: \"1a12882f-bc9e-40d3-a9f2-de5778150aaf\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" Jul 10 19:40:53.465110 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.464987 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a12882f-bc9e-40d3-a9f2-de5778150aaf-podres\") pod \"perf-node-gather-daemonset-jtl7s\" (UID: \"1a12882f-bc9e-40d3-a9f2-de5778150aaf\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" Jul 10 19:40:53.465110 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.465024 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a12882f-bc9e-40d3-a9f2-de5778150aaf-proc\") pod \"perf-node-gather-daemonset-jtl7s\" (UID: \"1a12882f-bc9e-40d3-a9f2-de5778150aaf\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" Jul 10 19:40:53.465267 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.465114 2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a12882f-bc9e-40d3-a9f2-de5778150aaf-lib-modules\") pod \"perf-node-gather-daemonset-jtl7s\" (UID: \"1a12882f-bc9e-40d3-a9f2-de5778150aaf\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" Jul 10 19:40:53.565864 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.565828 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a12882f-bc9e-40d3-a9f2-de5778150aaf-sys\") pod \"perf-node-gather-daemonset-jtl7s\" (UID: \"1a12882f-bc9e-40d3-a9f2-de5778150aaf\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" Jul 10 19:40:53.566325 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.566072 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a12882f-bc9e-40d3-a9f2-de5778150aaf-podres\") pod \"perf-node-gather-daemonset-jtl7s\" (UID: \"1a12882f-bc9e-40d3-a9f2-de5778150aaf\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" Jul 10 19:40:53.566325 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.565995 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1a12882f-bc9e-40d3-a9f2-de5778150aaf-sys\") pod \"perf-node-gather-daemonset-jtl7s\" (UID: \"1a12882f-bc9e-40d3-a9f2-de5778150aaf\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" Jul 10 19:40:53.566325 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.566127 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a12882f-bc9e-40d3-a9f2-de5778150aaf-proc\") pod \"perf-node-gather-daemonset-jtl7s\" (UID: \"1a12882f-bc9e-40d3-a9f2-de5778150aaf\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" Jul 10 19:40:53.566325 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.566182 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a12882f-bc9e-40d3-a9f2-de5778150aaf-lib-modules\") pod \"perf-node-gather-daemonset-jtl7s\" (UID: \"1a12882f-bc9e-40d3-a9f2-de5778150aaf\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" Jul 10 19:40:53.566325 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.566213 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1a12882f-bc9e-40d3-a9f2-de5778150aaf-podres\") pod \"perf-node-gather-daemonset-jtl7s\" (UID: \"1a12882f-bc9e-40d3-a9f2-de5778150aaf\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" Jul 10 19:40:53.566325 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.566232 2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmqtx\" (UniqueName: \"kubernetes.io/projected/1a12882f-bc9e-40d3-a9f2-de5778150aaf-kube-api-access-fmqtx\") pod \"perf-node-gather-daemonset-jtl7s\" (UID: \"1a12882f-bc9e-40d3-a9f2-de5778150aaf\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" Jul 10 19:40:53.566325 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.566300 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1a12882f-bc9e-40d3-a9f2-de5778150aaf-lib-modules\") pod \"perf-node-gather-daemonset-jtl7s\" (UID: \"1a12882f-bc9e-40d3-a9f2-de5778150aaf\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" Jul 10 19:40:53.566325 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.566310 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1a12882f-bc9e-40d3-a9f2-de5778150aaf-proc\") pod \"perf-node-gather-daemonset-jtl7s\" (UID: \"1a12882f-bc9e-40d3-a9f2-de5778150aaf\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" Jul 10 19:40:53.575605 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.575566 2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmqtx\" (UniqueName: \"kubernetes.io/projected/1a12882f-bc9e-40d3-a9f2-de5778150aaf-kube-api-access-fmqtx\") pod \"perf-node-gather-daemonset-jtl7s\" (UID: \"1a12882f-bc9e-40d3-a9f2-de5778150aaf\") " pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" Jul 10 19:40:53.749043 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.749010 2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" Jul 10 19:40:53.935220 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:53.935191 2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s"] Jul 10 19:40:53.937745 ip-10-0-130-74 kubenswrapper[2561]: W0710 19:40:53.937701 2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1a12882f_bc9e_40d3_a9f2_de5778150aaf.slice/crio-4bad91a0f61310a261b66eba06cd6141737ace3bef531547c74dc6d29a629da0 WatchSource:0}: Error finding container 4bad91a0f61310a261b66eba06cd6141737ace3bef531547c74dc6d29a629da0: Status 404 returned error can't find the container with id 4bad91a0f61310a261b66eba06cd6141737ace3bef531547c74dc6d29a629da0 Jul 10 19:40:54.762519 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:54.762470 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" event={"ID":"1a12882f-bc9e-40d3-a9f2-de5778150aaf","Type":"ContainerStarted","Data":"0e6b64c7daed034e5a2e8a35d3b11df1e4e62483d768b7626ffdd9a680ab8366"} Jul 10 19:40:54.763041 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:54.762551 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" Jul 10 19:40:54.763041 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:54.762580 2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" event={"ID":"1a12882f-bc9e-40d3-a9f2-de5778150aaf","Type":"ContainerStarted","Data":"4bad91a0f61310a261b66eba06cd6141737ace3bef531547c74dc6d29a629da0"} Jul 10 19:40:54.780893 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:54.780843 2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" podStartSLOduration=1.780828293 podStartE2EDuration="1.780828293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 19:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 19:40:54.779314225 +0000 UTC m=+2421.380456364" watchObservedRunningTime="2026-07-10 19:40:54.780828293 +0000 UTC m=+2421.381970456" Jul 10 19:40:56.626145 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:56.626096 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hlcx4_a9626c55-cbe1-411e-9996-31d97196f235/dns/0.log" Jul 10 19:40:56.695629 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:56.695600 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hlcx4_a9626c55-cbe1-411e-9996-31d97196f235/kube-rbac-proxy/0.log" Jul 10 19:40:56.841726 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:56.841694 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gp2jn_452ded98-132f-4e28-865a-d2f55032625c/dns-node-resolver/0.log" Jul 10 19:40:57.567754 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:57.567723 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jb7v6_ffe36bf2-7d7e-426f-a273-9acdbd6935e4/node-ca/0.log" Jul 10 19:40:58.724482 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:58.724449 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b4fc6b9d-7s88n_36c31443-af47-4c59-a1d4-a00aa0f29dee/kube-auth-proxy/0.log" Jul 10 19:40:59.721715 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:40:59.721688 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-92mhk_0d117dad-109a-41fc-a1f0-0530aff4279e/serve-healthcheck-canary/0.log" Jul 10 19:41:00.396099 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:41:00.396067 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rlz4g_503b2049-91bb-4f7f-a230-6ec6653c6b62/kube-rbac-proxy/0.log" Jul 10 19:41:00.420758 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:41:00.420720 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rlz4g_503b2049-91bb-4f7f-a230-6ec6653c6b62/exporter/0.log" Jul 10 19:41:00.446629 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:41:00.446600 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rlz4g_503b2049-91bb-4f7f-a230-6ec6653c6b62/extractor/0.log" Jul 10 19:41:00.782798 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:41:00.782127 2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-z44r6/perf-node-gather-daemonset-jtl7s" Jul 10 19:41:03.541230 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:41:03.541151 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7668cd484b-9qvkc_22acb39a-d928-4715-a844-945c72839563/manager/0.log" Jul 10 19:41:03.566795 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:41:03.566762 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7668cd484b-9qvkc_22acb39a-d928-4715-a844-945c72839563/copy-manifests/0.log" Jul 10 19:41:04.168087 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:41:04.168056 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b4fc6b9d-7s88n_36c31443-af47-4c59-a1d4-a00aa0f29dee/kube-auth-proxy/0.log" Jul 10 19:41:04.947920 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:41:04.947894 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6d9b94cf9d-lbwrk_c87cd792-cd69-47b3-bebe-213827cbb39d/manager/0.log" Jul 10 19:41:04.975604 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:41:04.975568 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-gzgx6_b234f450-466c-46b3-9380-556e8109a117/openshift-lws-operator/0.log" Jul 10 19:41:05.725614 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:41:05.725531 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b4fc6b9d-7s88n_36c31443-af47-4c59-a1d4-a00aa0f29dee/kube-auth-proxy/0.log" Jul 10 19:41:09.917144 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:41:09.917106 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-nfcpl_06cdb497-0536-4b4e-99b1-b950067a5fa8/migrator/0.log" Jul 10 19:41:09.940454 ip-10-0-130-74 kubenswrapper[2561]: I0710 19:41:09.940425 2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-nfcpl_06cdb497-0536-4b4e-99b1-b950067a5fa8/graceful-termination/0.log"