Jul 07 14:24:36.072441 ip-10-0-128-34 systemd[1]: Starting Kubernetes Kubelet... Jul 07 14:24:36.463824 ip-10-0-128-34 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 14:24:36.463824 ip-10-0-128-34 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 07 14:24:36.463824 ip-10-0-128-34 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 14:24:36.463824 ip-10-0-128-34 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 07 14:24:36.463824 ip-10-0-128-34 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 07 14:24:36.464544 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.463888 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 07 14:24:36.469622 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469601 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:24:36.469622 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469620 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469624 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469629 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469632 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469635 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469638 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469641 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469643 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469646 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469649 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469652 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469655 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469657 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469660 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469663 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469666 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469668 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469679 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469682 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469684 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:24:36.469691 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469687 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469689 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469692 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469695 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469698 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469701 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469704 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469707 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469710 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469712 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469715 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469717 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469720 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469722 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469725 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469728 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469730 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469733 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469735 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469738 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:24:36.470167 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469740 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469742 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469745 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469747 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469750 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469752 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469755 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469757 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469760 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469762 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469767 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469769 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469772 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469774 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469777 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469780 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469783 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469785 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469788 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469790 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:24:36.470652 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469793 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:24:36.471124 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469795 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:24:36.471124 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469798 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:24:36.471124 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469800 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:24:36.471124 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469803 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:24:36.471124 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469806 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:24:36.471124 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469809 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:24:36.471124 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469812 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:24:36.471124 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469816 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:24:36.471124 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469820 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:24:36.471124 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469824 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:24:36.471124 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469827 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:24:36.471124 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469830 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:24:36.471124 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469833 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:24:36.471124 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469836 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:24:36.471124 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469839 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:24:36.471124 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469842 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:24:36.471124 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469846 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:24:36.471124 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469848 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:24:36.471124 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469851 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:24:36.471612 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469854 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:24:36.471612 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469857 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:24:36.471612 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469861 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:24:36.471612 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469865 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:24:36.471612 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469868 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:24:36.471612 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469872 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:24:36.471612 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469876 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:24:36.471612 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469878 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:24:36.471612 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469881 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:24:36.471612 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469883 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:24:36.471612 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469886 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:24:36.471612 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469889 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:24:36.471612 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469892 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:24:36.471612 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469895 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:24:36.471612 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469897 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:24:36.471612 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469900 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:24:36.471612 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469902 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:24:36.471612 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.469905 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:24:36.471612 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470628 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:24:36.472102 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470635 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:24:36.472102 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470640 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:24:36.472102 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470644 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:24:36.472102 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470648 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:24:36.472102 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470651 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:24:36.472102 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470654 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:24:36.472102 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470657 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:24:36.472102 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470660 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:24:36.472102 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470662 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:24:36.472102 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470665 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:24:36.472102 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470668 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:24:36.472102 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470672 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:24:36.472102 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470676 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:24:36.472102 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470679 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:24:36.472102 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470681 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:24:36.472102 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470684 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:24:36.472102 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470687 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:24:36.472102 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470690 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:24:36.472102 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470694 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470697 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470700 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470703 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470705 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470708 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470710 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470713 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470716 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470718 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470721 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470724 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470726 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470729 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470731 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470734 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470737 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470739 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470742 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470744 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:24:36.472583 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470747 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470749 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470753 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470755 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470758 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470760 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470763 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470765 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470768 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470771 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470774 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470776 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470780 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470783 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470786 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470788 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470791 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470794 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470796 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470799 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:24:36.473063 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470801 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:24:36.473568 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470804 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:24:36.473568 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470810 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:24:36.473568 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470813 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:24:36.473568 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470815 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:24:36.473568 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470818 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:24:36.473568 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470821 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:24:36.473568 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470823 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:24:36.473568 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470826 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:24:36.473568 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470828 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:24:36.473568 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470831 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:24:36.473568 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470833 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:24:36.473568 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470836 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:24:36.473568 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470838 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:24:36.473568 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470841 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:24:36.473568 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470844 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:24:36.473568 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470847 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:24:36.473568 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470850 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:24:36.473568 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470852 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:24:36.473568 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470855 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470857 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470860 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470863 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470865 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470868 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470879 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470882 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470885 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470887 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470890 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470892 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470895 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470898 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470901 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470904 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470906 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470909 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470911 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470914 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:24:36.474047 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.470916 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471348 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471363 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471367 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471382 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471386 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471390 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471393 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471396 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471399 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471401 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471404 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471407 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471410 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471412 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471415 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471418 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471421 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471423 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471426 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:24:36.474539 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471429 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:24:36.475024 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471431 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:24:36.475024 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471434 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:24:36.475024 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471436 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:24:36.475024 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471439 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:24:36.475024 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471442 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:24:36.475024 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471445 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:24:36.475024 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471447 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:24:36.475024 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471451 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:24:36.475024 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471455 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:24:36.475024 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471458 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:24:36.475024 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471461 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:24:36.475024 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471464 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:24:36.475024 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471466 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:24:36.475024 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471472 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:24:36.475024 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471476 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:24:36.475024 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471479 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:24:36.475024 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471482 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:24:36.475024 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471484 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:24:36.475024 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471487 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471490 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471492 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471495 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471497 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471500 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471503 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471505 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471509 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471511 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471514 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471517 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471519 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471522 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471525 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471527 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471530 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471533 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471535 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471539 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:24:36.475484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471542 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471544 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471547 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471550 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471552 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471556 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471559 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471563 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471566 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471569 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471571 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471574 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471577 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471579 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471582 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471585 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471587 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471590 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471593 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471596 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:24:36.475976 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471598 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471601 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471603 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471606 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471608 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471611 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471613 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471616 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471619 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471622 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471625 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471628 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471631 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471633 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471636 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471638 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471641 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471644 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471646 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471649 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:24:36.476476 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.471652 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471728 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471735 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471744 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471748 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471753 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471756 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471760 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471765 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471768 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471771 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471775 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471778 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471782 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471785 2562 flags.go:64] FLAG: --cgroup-root="" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471788 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471791 2562 flags.go:64] FLAG: --client-ca-file="" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471794 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471797 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471802 2562 flags.go:64] FLAG: --cluster-domain="" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471804 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471808 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471811 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471815 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 07 14:24:36.476985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471819 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471822 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471825 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471828 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471831 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471834 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471837 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471841 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471844 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471848 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471852 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471855 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471858 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471861 2562 flags.go:64] FLAG: --enable-server="true" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471864 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471869 2562 flags.go:64] FLAG: --event-burst="100" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471873 2562 flags.go:64] FLAG: --event-qps="50" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471875 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471878 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471881 2562 flags.go:64] FLAG: --eviction-hard="" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471885 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471888 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471891 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471894 2562 flags.go:64] FLAG: --eviction-soft="" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471897 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 07 14:24:36.477569 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471900 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471903 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471906 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471909 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471911 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471915 2562 flags.go:64] FLAG: --feature-gates="" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471919 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471922 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471925 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471928 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471931 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471934 2562 flags.go:64] FLAG: --help="false" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471937 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-128-34.ec2.internal" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471941 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471944 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471947 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471950 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471954 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471958 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471960 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471964 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471967 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471970 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471973 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 07 14:24:36.478279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471976 2562 flags.go:64] FLAG: --kube-reserved="" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471979 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471982 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471985 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471988 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471990 2562 flags.go:64] FLAG: --lock-file="" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471993 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471996 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.471999 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472005 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472008 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472011 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472014 2562 flags.go:64] FLAG: --logging-format="text" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472017 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472020 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472023 2562 flags.go:64] FLAG: --manifest-url="" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472026 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472031 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472033 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472038 2562 flags.go:64] FLAG: --max-pods="110" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472041 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472044 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472048 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472051 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472055 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 07 14:24:36.478884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472058 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472061 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472069 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472072 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472075 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472078 2562 flags.go:64] FLAG: --pod-cidr="" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472081 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472086 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472089 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472093 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472096 2562 flags.go:64] FLAG: --port="10250" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472099 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472101 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00db5eee7b0fa156d" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472105 2562 flags.go:64] FLAG: --qos-reserved="" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472108 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472111 2562 flags.go:64] FLAG: --register-node="true" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472114 2562 flags.go:64] FLAG: --register-with-taints="" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472118 2562 flags.go:64] FLAG: --registry-burst="10" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472121 2562 flags.go:64] FLAG: --registry-qps="5" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472124 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472130 2562 flags.go:64] FLAG: --reserved-memory="" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472134 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472137 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472140 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472143 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 07 14:24:36.479788 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472146 2562 flags.go:64] FLAG: --runonce="false" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472149 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472152 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472155 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472158 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472161 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472164 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472168 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472171 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472174 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472177 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472180 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472183 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472186 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472189 2562 flags.go:64] FLAG: --system-cgroups="" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472192 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472197 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472200 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472203 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472207 2562 flags.go:64] FLAG: --tls-min-version="" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472209 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472212 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472215 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472218 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472221 2562 flags.go:64] FLAG: --v="2" Jul 07 14:24:36.480689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472225 2562 flags.go:64] FLAG: --version="false" Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472229 2562 flags.go:64] FLAG: --vmodule="" Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472234 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.472238 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472354 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472358 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472361 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472363 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472366 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472368 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472384 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472386 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472391 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472393 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472396 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472399 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472402 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472404 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472408 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472411 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472413 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:24:36.481344 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472416 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:24:36.481905 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472419 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:24:36.481905 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472421 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:24:36.481905 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472424 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:24:36.481905 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472427 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:24:36.481905 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472430 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:24:36.481905 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472433 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:24:36.481905 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472435 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:24:36.481905 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472438 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:24:36.481905 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472440 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:24:36.481905 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472443 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:24:36.481905 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472446 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:24:36.481905 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472448 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:24:36.481905 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472452 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:24:36.481905 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472456 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:24:36.481905 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472461 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:24:36.481905 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472464 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:24:36.481905 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472467 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:24:36.481905 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472469 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:24:36.481905 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472473 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:24:36.482386 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472475 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:24:36.482386 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472478 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:24:36.482386 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472480 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:24:36.482386 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472486 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:24:36.482386 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472489 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:24:36.482386 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472491 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:24:36.482386 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472494 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:24:36.482386 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472497 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:24:36.482386 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472500 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:24:36.482386 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472503 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:24:36.482386 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472506 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:24:36.482386 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472510 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:24:36.482386 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472514 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:24:36.482386 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472517 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:24:36.482386 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472520 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:24:36.482386 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472523 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:24:36.482386 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472526 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:24:36.482386 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472530 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:24:36.482386 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472532 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472535 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472538 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472540 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472543 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472546 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472548 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472552 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472555 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472558 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472560 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472563 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472565 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472568 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472571 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472573 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472576 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472580 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472583 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472586 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:24:36.482846 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472588 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472591 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472594 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472597 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472599 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472603 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472605 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472608 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472610 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472613 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472616 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472618 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472621 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472624 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472626 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472629 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472631 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472634 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472637 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472641 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:24:36.483334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472644 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:24:36.483839 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472647 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:24:36.483839 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472649 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:24:36.483839 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.472652 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:24:36.483839 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.473291 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 14:24:36.483839 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.481459 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 07 14:24:36.483839 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.481480 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 07 14:24:36.483839 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.482182 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 07 14:24:36.483839 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.482189 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 07 14:24:36.483839 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482249 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:24:36.483839 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482255 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:24:36.483839 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482258 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:24:36.483839 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482262 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:24:36.483839 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482266 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:24:36.483839 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482269 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:24:36.483839 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482271 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482274 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482277 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482280 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482282 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482285 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482288 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482290 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482293 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482296 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482304 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482307 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482310 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482313 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482316 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482319 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482322 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482325 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482328 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482331 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:24:36.484217 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482334 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:24:36.484710 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482336 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:24:36.484710 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482340 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:24:36.484710 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482343 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:24:36.484710 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482346 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:24:36.484710 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482349 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:24:36.484710 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482351 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:24:36.484710 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482354 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:24:36.484710 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482357 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:24:36.484710 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482360 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:24:36.484710 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482363 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:24:36.484710 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482366 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:24:36.484710 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482368 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:24:36.484710 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482386 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:24:36.484710 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482390 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:24:36.484710 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482393 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:24:36.484710 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482398 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:24:36.484710 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482402 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:24:36.484710 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482405 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:24:36.484710 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482408 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482412 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482415 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482418 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482421 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482423 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482426 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482429 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482431 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482435 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482437 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482440 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482443 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482446 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482448 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482451 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482454 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482457 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482460 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482463 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:24:36.485180 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482465 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:24:36.485680 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482468 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:24:36.485680 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482471 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:24:36.485680 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482473 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:24:36.485680 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482476 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:24:36.485680 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482478 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:24:36.485680 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482481 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:24:36.485680 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482484 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:24:36.485680 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482486 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:24:36.485680 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482489 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:24:36.485680 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482491 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:24:36.485680 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482494 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:24:36.485680 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482496 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:24:36.485680 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482499 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:24:36.485680 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482505 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:24:36.485680 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482507 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:24:36.485680 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482511 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:24:36.485680 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482515 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:24:36.485680 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482518 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:24:36.485680 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482522 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:24:36.486149 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482526 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:24:36.486149 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482530 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:24:36.486149 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482533 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:24:36.486149 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482537 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:24:36.486149 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482540 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:24:36.486149 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482543 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:24:36.486149 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482546 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:24:36.486149 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482548 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:24:36.486149 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482551 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:24:36.486149 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482554 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:24:36.486149 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482557 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:24:36.486149 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482559 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:24:36.486149 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482562 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:24:36.486149 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482564 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:24:36.486565 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.482569 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 14:24:36.486565 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482687 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 07 14:24:36.486565 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482692 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 07 14:24:36.486565 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482695 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 07 14:24:36.486565 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482698 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 07 14:24:36.486565 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482701 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 07 14:24:36.486565 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482704 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 07 14:24:36.486565 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482707 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 07 14:24:36.486565 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482710 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 07 14:24:36.486565 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482713 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 07 14:24:36.486565 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482716 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 07 14:24:36.486565 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482720 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 07 14:24:36.486565 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482724 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 07 14:24:36.486565 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482728 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482731 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482735 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482738 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482741 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482744 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482748 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482750 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482753 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482756 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482759 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482762 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482765 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482767 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482770 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482773 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482775 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482778 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482780 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482783 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 07 14:24:36.486975 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482786 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 07 14:24:36.487480 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482788 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 07 14:24:36.487480 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482791 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 07 14:24:36.487480 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482793 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 07 14:24:36.487480 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482796 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 07 14:24:36.487480 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482798 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 07 14:24:36.487480 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482801 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 07 14:24:36.487480 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482803 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 07 14:24:36.487480 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482806 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 07 14:24:36.487480 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482808 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 07 14:24:36.487480 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482811 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 07 14:24:36.487480 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482813 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 07 14:24:36.487480 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482816 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 07 14:24:36.487480 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482819 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 07 14:24:36.487480 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482822 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 07 14:24:36.487480 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482825 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 07 14:24:36.487480 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482827 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 07 14:24:36.487480 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482830 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 07 14:24:36.487480 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482833 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 07 14:24:36.487480 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482836 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 07 14:24:36.487951 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482839 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 07 14:24:36.487951 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482841 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 07 14:24:36.487951 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482844 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 07 14:24:36.487951 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482847 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 07 14:24:36.487951 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482849 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 07 14:24:36.487951 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482852 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 07 14:24:36.487951 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482855 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 07 14:24:36.487951 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482859 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 07 14:24:36.487951 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482863 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 07 14:24:36.487951 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482866 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 07 14:24:36.487951 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482869 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 07 14:24:36.487951 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482872 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 07 14:24:36.487951 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482875 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 07 14:24:36.487951 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482877 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 07 14:24:36.487951 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482880 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 07 14:24:36.487951 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482882 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 07 14:24:36.487951 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482885 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 07 14:24:36.487951 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482887 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 07 14:24:36.487951 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482890 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482892 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482895 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482897 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482900 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482902 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482905 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482907 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482911 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482913 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482916 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482919 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482922 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482924 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482927 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482930 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482932 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482934 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482937 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482939 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 07 14:24:36.488420 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482942 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 07 14:24:36.488916 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482944 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 07 14:24:36.488916 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482947 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 07 14:24:36.488916 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482949 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 07 14:24:36.488916 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482952 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 07 14:24:36.488916 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482955 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 07 14:24:36.488916 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482957 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 07 14:24:36.488916 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482960 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 07 14:24:36.488916 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:36.482962 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 07 14:24:36.488916 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.482967 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 07 14:24:36.488916 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.483094 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 07 14:24:36.488916 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.486171 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 07 14:24:36.488916 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.487422 2562 server.go:1019] "Starting client certificate rotation" Jul 07 14:24:36.488916 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.487534 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 14:24:36.488916 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.487580 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 07 14:24:36.489858 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.489842 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 14:24:36.491043 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.491024 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 07 14:24:36.499528 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.499506 2562 log.go:25] "Validated CRI v1 runtime API" Jul 07 14:24:36.501538 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.501521 2562 log.go:25] "Validated CRI v1 image API" Jul 07 14:24:36.502340 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.502322 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 07 14:24:36.505913 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.505886 2562 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 923f32f5-f3fb-4ea2-b46d-98ce1904a3c9:/dev/nvme0n1p4 9e8c584f-7db9-45d8-b4ca-acad67ac0db6:/dev/nvme0n1p3] Jul 07 14:24:36.505993 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.505911 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 07 14:24:36.510810 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.510694 2562 manager.go:216] Machine: {Timestamp:2026-07-07 14:24:36.510112647 +0000 UTC m=+0.319660127 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093860 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29ff83dd0cbcaff77c132c257a8ac1 SystemUUID:ec29ff83-dd0c-bcaf-f77c-132c257a8ac1 BootID:bd1b32f0-3cc2-470d-9189-c8cfaf6e098f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f5:99:87:c8:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f5:99:87:c8:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:ae:88:46:91:48 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 07 14:24:36.510810 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.510803 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 07 14:24:36.510948 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.510936 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 07 14:24:36.512577 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.512547 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 07 14:24:36.512750 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.512581 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-34.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 07 14:24:36.513334 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.513321 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 07 14:24:36.513407 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.513340 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 07 14:24:36.513407 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.513351 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 14:24:36.513407 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.513366 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 07 14:24:36.514620 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.514607 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 07 14:24:36.517637 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.517620 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 07 14:24:36.517781 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.517770 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 07 14:24:36.517838 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.517809 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 07 14:24:36.517838 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.517828 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 07 14:24:36.517927 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.517848 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 07 14:24:36.517927 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.517865 2562 kubelet.go:398] "Adding apiserver pod source" Jul 07 14:24:36.517927 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.517889 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 07 14:24:36.519814 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.519703 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 14:24:36.520771 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.520751 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 14:24:36.520838 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.520795 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 07 14:24:36.523789 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.523775 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 07 14:24:36.525661 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.525647 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 07 14:24:36.525700 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.525678 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 07 14:24:36.526657 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.526646 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 07 14:24:36.526702 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.526664 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 07 14:24:36.526702 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.526670 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 07 14:24:36.526702 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.526675 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 07 14:24:36.526702 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.526684 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 07 14:24:36.526702 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.526689 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 07 14:24:36.526702 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.526697 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 07 14:24:36.526702 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.526702 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 07 14:24:36.526869 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.526709 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 07 14:24:36.526869 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.526714 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 07 14:24:36.526869 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.526723 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 07 14:24:36.526869 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.526728 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 07 14:24:36.527512 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.527498 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 07 14:24:36.527553 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.527514 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 07 14:24:36.530616 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.530603 2562 server.go:1268] "Started kubelet" Jul 07 14:24:36.530740 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.530702 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 07 14:24:36.530795 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.530676 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 07 14:24:36.530840 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.530794 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 07 14:24:36.531688 ip-10-0-128-34 systemd[1]: Started Kubernetes Kubelet. Jul 07 14:24:36.532428 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.532412 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 07 14:24:36.532886 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.532873 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 07 14:24:36.533647 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.533626 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-34.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 07 14:24:36.533818 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:36.533799 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 07 14:24:36.534070 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:36.534050 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-34.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 07 14:24:36.538335 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:36.538305 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 07 14:24:36.538815 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.538800 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 07 14:24:36.538871 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.538799 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 07 14:24:36.539002 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:36.538982 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-34.ec2.internal\" not found" Jul 07 14:24:36.539191 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:36.538281 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-34.ec2.internal.18c0080684dc52e2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-34.ec2.internal,UID:ip-10-0-128-34.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-34.ec2.internal,},FirstTimestamp:2026-07-07 14:24:36.53057405 +0000 UTC m=+0.340121531,LastTimestamp:2026-07-07 14:24:36.53057405 +0000 UTC m=+0.340121531,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-34.ec2.internal,}" Jul 07 14:24:36.539453 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.539434 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 07 14:24:36.539453 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.539435 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 07 14:24:36.539570 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.539466 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 07 14:24:36.539570 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.539561 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 07 14:24:36.539570 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.539568 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 07 14:24:36.539973 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.539786 2562 factory.go:55] Registering systemd factory Jul 07 14:24:36.539973 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.539800 2562 factory.go:223] Registration of the systemd container factory successfully Jul 07 14:24:36.540067 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.540053 2562 factory.go:153] Registering CRI-O factory Jul 07 14:24:36.540112 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.540072 2562 factory.go:223] Registration of the crio container factory successfully Jul 07 14:24:36.540156 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.540125 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 07 14:24:36.540156 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.540146 2562 factory.go:103] Registering Raw factory Jul 07 14:24:36.540239 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.540158 2562 manager.go:1196] Started watching for new ooms in manager Jul 07 14:24:36.540615 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.540597 2562 manager.go:349] Starting recovery of all containers Jul 07 14:24:36.544981 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:36.544946 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 07 14:24:36.545114 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:36.545089 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-34.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 07 14:24:36.547043 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.547015 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2mtjp" Jul 07 14:24:36.553050 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.553033 2562 manager.go:354] Recovery completed Jul 07 14:24:36.557481 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.557466 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:24:36.558976 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.558959 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2mtjp" Jul 07 14:24:36.560200 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.560176 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:24:36.560290 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.560217 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:24:36.560290 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.560232 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:24:36.561337 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.561322 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 07 14:24:36.561337 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.561335 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 07 14:24:36.561337 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.561335 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:24:36.561544 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.561352 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 07 14:24:36.562126 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.562109 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:24:36.562195 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.562132 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:24:36.562195 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.562143 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:24:36.562266 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:36.562199 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-34.ec2.internal.18c0080686a06181 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-34.ec2.internal,UID:ip-10-0-128-34.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-34.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-34.ec2.internal,},FirstTimestamp:2026-07-07 14:24:36.560200065 +0000 UTC m=+0.369747547,LastTimestamp:2026-07-07 14:24:36.560200065 +0000 UTC m=+0.369747547,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-34.ec2.internal,}" Jul 07 14:24:36.562420 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.562408 2562 policy_none.go:49] "None policy: Start" Jul 07 14:24:36.562474 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.562427 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 07 14:24:36.562474 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.562443 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 07 14:24:36.563448 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.563435 2562 policy_none.go:47] "Start" Jul 07 14:24:36.601034 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.601012 2562 manager.go:337] "Starting Device Plugin manager" Jul 07 14:24:36.601133 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:36.601052 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 07 14:24:36.601133 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.601067 2562 server.go:87] "Starting device plugin registration server" Jul 07 14:24:36.601530 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.601355 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 07 14:24:36.601609 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.601536 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 07 14:24:36.601668 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.601648 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 07 14:24:36.601757 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.601741 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 07 14:24:36.601757 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.601754 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 07 14:24:36.602687 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:36.602667 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 07 14:24:36.602768 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:36.602724 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-34.ec2.internal\" not found" Jul 07 14:24:36.698809 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.698768 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 07 14:24:36.700195 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.700175 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 07 14:24:36.700274 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.700230 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 07 14:24:36.700274 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.700262 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 07 14:24:36.700355 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:36.700308 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 07 14:24:36.701677 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.701656 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:24:36.702572 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.702548 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:24:36.702663 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.702598 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:24:36.702663 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.702615 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:24:36.702663 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.702654 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-34.ec2.internal" Jul 07 14:24:36.703536 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.703512 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:24:36.712269 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.712251 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-34.ec2.internal" Jul 07 14:24:36.712349 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:36.712277 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-34.ec2.internal\": node \"ip-10-0-128-34.ec2.internal\" not found" Jul 07 14:24:36.732237 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:36.732139 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-34.ec2.internal\" not found" Jul 07 14:24:36.803823 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.803764 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-34.ec2.internal"] Jul 07 14:24:36.804019 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.803873 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:24:36.805051 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.805032 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:24:36.805162 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.805065 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:24:36.805162 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.805082 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:24:36.806797 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.806783 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:24:36.806948 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.806933 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal" Jul 07 14:24:36.807005 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.806964 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:24:36.807644 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.807625 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:24:36.807722 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.807657 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:24:36.807722 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.807666 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:24:36.807722 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.807672 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:24:36.807722 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.807684 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:24:36.807722 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.807699 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:24:36.809162 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.809144 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.ec2.internal" Jul 07 14:24:36.809219 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.809174 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 07 14:24:36.809905 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.809889 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasSufficientMemory" Jul 07 14:24:36.809962 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.809915 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasNoDiskPressure" Jul 07 14:24:36.809962 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.809930 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeHasSufficientPID" Jul 07 14:24:36.832286 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:36.832257 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-34.ec2.internal\" not found" Jul 07 14:24:36.834468 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:36.834431 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-34.ec2.internal\" not found" node="ip-10-0-128-34.ec2.internal" Jul 07 14:24:36.839325 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:36.839306 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-34.ec2.internal\" not found" node="ip-10-0-128-34.ec2.internal" Jul 07 14:24:36.932728 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:36.932684 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-34.ec2.internal\" not found" Jul 07 14:24:36.948136 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.948094 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f5c9d3f2e625fd40d222604d069d6f5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal\" (UID: \"4f5c9d3f2e625fd40d222604d069d6f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal" Jul 07 14:24:36.948136 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.948139 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f5c9d3f2e625fd40d222604d069d6f5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal\" (UID: \"4f5c9d3f2e625fd40d222604d069d6f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal" Jul 07 14:24:36.948335 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:36.948165 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b2a01d40505439c955139cb876044b6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-34.ec2.internal\" (UID: \"2b2a01d40505439c955139cb876044b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.ec2.internal" Jul 07 14:24:37.033654 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:37.033565 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-34.ec2.internal\" not found" Jul 07 14:24:37.049129 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:37.049100 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b2a01d40505439c955139cb876044b6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-34.ec2.internal\" (UID: \"2b2a01d40505439c955139cb876044b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.ec2.internal" Jul 07 14:24:37.049193 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:37.049132 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f5c9d3f2e625fd40d222604d069d6f5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal\" (UID: \"4f5c9d3f2e625fd40d222604d069d6f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal" Jul 07 14:24:37.049193 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:37.049151 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f5c9d3f2e625fd40d222604d069d6f5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal\" (UID: \"4f5c9d3f2e625fd40d222604d069d6f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal" Jul 07 14:24:37.049257 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:37.049200 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f5c9d3f2e625fd40d222604d069d6f5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal\" (UID: \"4f5c9d3f2e625fd40d222604d069d6f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal" Jul 07 14:24:37.049257 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:37.049218 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2b2a01d40505439c955139cb876044b6-config\") pod \"kube-apiserver-proxy-ip-10-0-128-34.ec2.internal\" (UID: \"2b2a01d40505439c955139cb876044b6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.ec2.internal" Jul 07 14:24:37.049257 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:37.049227 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4f5c9d3f2e625fd40d222604d069d6f5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal\" (UID: \"4f5c9d3f2e625fd40d222604d069d6f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal" Jul 07 14:24:37.138807 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:37.138758 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-34.ec2.internal\" not found" Jul 07 14:24:37.138807 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:37.138785 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal" Jul 07 14:24:37.143186 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:37.142919 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.ec2.internal" Jul 07 14:24:37.239541 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:37.239492 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-34.ec2.internal\" not found" Jul 07 14:24:37.339870 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:37.339783 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-34.ec2.internal\" not found" Jul 07 14:24:37.440484 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:37.440436 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-34.ec2.internal\" not found" Jul 07 14:24:37.488966 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:37.488929 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 07 14:24:37.489472 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:37.489096 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 07 14:24:37.541249 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:37.541209 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-34.ec2.internal\" not found" Jul 07 14:24:37.544325 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:37.544306 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:24:37.562040 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:37.561996 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 14:19:37 +0000 UTC" deadline="2028-03-11 03:21:57.255853885 +0000 UTC" Jul 07 14:24:37.562040 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:37.562029 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14700h57m19.693828471s" Jul 07 14:24:37.639216 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:37.639129 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal" Jul 07 14:24:37.657922 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:37.657891 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 14:24:37.659646 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:37.659631 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.ec2.internal" Jul 07 14:24:37.672068 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:37.672041 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 07 14:24:37.835061 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:37.835029 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:24:37.863140 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:37.863093 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2a01d40505439c955139cb876044b6.slice/crio-d06ef322cc2c1547d1529bc0613634db18d63fb9b8fb5755697fb94d9a6efcfc WatchSource:0}: Error finding container d06ef322cc2c1547d1529bc0613634db18d63fb9b8fb5755697fb94d9a6efcfc: Status 404 returned error can't find the container with id d06ef322cc2c1547d1529bc0613634db18d63fb9b8fb5755697fb94d9a6efcfc Jul 07 14:24:37.864336 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:37.864296 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5c9d3f2e625fd40d222604d069d6f5.slice/crio-e09b81106106144cd853f6a795ee769f1de7d40f19d76cf99822446f8df94623 WatchSource:0}: Error finding container e09b81106106144cd853f6a795ee769f1de7d40f19d76cf99822446f8df94623: Status 404 returned error can't find the container with id e09b81106106144cd853f6a795ee769f1de7d40f19d76cf99822446f8df94623 Jul 07 14:24:37.868112 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:37.868095 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:24:38.114274 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.114173 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:24:38.518855 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.518769 2562 apiserver.go:52] "Watching apiserver" Jul 07 14:24:38.528084 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.528051 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 07 14:24:38.528559 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.528528 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-34.ec2.internal","openshift-cluster-node-tuning-operator/tuned-7frb5","openshift-dns/node-resolver-8xs5n","openshift-image-registry/node-ca-zc4j9","openshift-multus/multus-5gg2b","openshift-multus/multus-additional-cni-plugins-d74p5","openshift-ovn-kubernetes/ovnkube-node-zzfcn","kube-system/global-pull-secret-syncer-7w875","kube-system/konnectivity-agent-88g99","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal","openshift-multus/network-metrics-daemon-4bgbc","openshift-network-diagnostics/network-check-target-6b55m","openshift-network-operator/iptables-alerter-qch8s"] Jul 07 14:24:38.530462 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.530439 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.532228 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.532200 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:24:38.532343 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:38.532285 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:24:38.533482 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.533451 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8xs5n" Jul 07 14:24:38.535195 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.534812 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zc4j9" Jul 07 14:24:38.536757 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.535952 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9l67r\"" Jul 07 14:24:38.536757 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.536413 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 07 14:24:38.536757 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.536637 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 07 14:24:38.536989 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.536849 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 07 14:24:38.537228 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.537204 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 07 14:24:38.538047 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.537679 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 07 14:24:38.538047 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.537972 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 07 14:24:38.539134 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.539112 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.539646 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.539513 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 07 14:24:38.540950 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.540766 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 07 14:24:38.541259 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.541182 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.542130 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.541965 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vmv7h\"" Jul 07 14:24:38.542130 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.542007 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 07 14:24:38.543888 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.543854 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 07 14:24:38.544100 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.544076 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 07 14:24:38.544682 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.544209 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 07 14:24:38.544682 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.544348 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mh8jq\"" Jul 07 14:24:38.544682 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.544376 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 07 14:24:38.544682 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.544508 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-klhtc\"" Jul 07 14:24:38.544939 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.544824 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-88g99" Jul 07 14:24:38.545877 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.545336 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 07 14:24:38.545877 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.545425 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 07 14:24:38.545877 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.545562 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 07 14:24:38.545877 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.545656 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 07 14:24:38.545877 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.545768 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xksww\"" Jul 07 14:24:38.547780 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.547422 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qch8s" Jul 07 14:24:38.548824 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.548659 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 07 14:24:38.549164 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.549132 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 07 14:24:38.549356 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.549297 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 07 14:24:38.549482 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.549424 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7czgm\"" Jul 07 14:24:38.550026 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.549978 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v5pdp\"" Jul 07 14:24:38.551890 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.551868 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 07 14:24:38.552109 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.552089 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:24:38.552691 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.552669 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.553530 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.553356 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 07 14:24:38.554839 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.554820 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.556011 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.555960 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-kubelet\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.556011 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.555998 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-var-lib-openvswitch\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.556170 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556024 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-run-ovn\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.556170 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556049 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b22731b9-be09-4ad5-96ae-dce04203e0ad-host\") pod \"node-ca-zc4j9\" (UID: \"b22731b9-be09-4ad5-96ae-dce04203e0ad\") " pod="openshift-image-registry/node-ca-zc4j9" Jul 07 14:24:38.556170 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556084 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-multus-conf-dir\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.556170 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556105 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-cnibin\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.556170 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556152 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-systemd-units\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.556170 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556173 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-run-netns\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.556525 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556194 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-etc-openvswitch\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.556525 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556219 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-run-openvswitch\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.556525 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556246 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-run-ovn-kubernetes\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.556525 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556270 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvkk7\" (UniqueName: \"kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7\") pod \"network-check-target-6b55m\" (UID: \"337cda64-fcd6-481c-86ed-b25d2396aaca\") " pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:24:38.556525 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556295 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-host-run-k8s-cni-cncf-io\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.556525 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556321 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/68513391-0bac-4ef4-bf7a-2488aea5b76c-multus-daemon-config\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.556525 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556348 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.556525 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556375 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64b3ea37-09bc-491f-8e4c-de9cc97e7b5a-hosts-file\") pod \"node-resolver-8xs5n\" (UID: \"64b3ea37-09bc-491f-8e4c-de9cc97e7b5a\") " pod="openshift-dns/node-resolver-8xs5n" Jul 07 14:24:38.556525 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556434 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64b3ea37-09bc-491f-8e4c-de9cc97e7b5a-tmp-dir\") pod \"node-resolver-8xs5n\" (UID: \"64b3ea37-09bc-491f-8e4c-de9cc97e7b5a\") " pod="openshift-dns/node-resolver-8xs5n" Jul 07 14:24:38.556525 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556458 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-host-var-lib-cni-multus\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.556525 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556484 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.556525 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556526 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-cni-netd\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.557121 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556609 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/48af10ba-c8eb-42d6-a322-440ed4e09d81-ovnkube-config\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.557121 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556643 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/48af10ba-c8eb-42d6-a322-440ed4e09d81-env-overrides\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.557121 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556702 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/48af10ba-c8eb-42d6-a322-440ed4e09d81-ovnkube-script-lib\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.557121 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556740 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:38.557121 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556753 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-host-run-netns\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.557121 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556782 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.557121 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:38.556815 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:24:38.557121 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556837 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/48af10ba-c8eb-42d6-a322-440ed4e09d81-ovn-node-metrics-cert\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.557121 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556869 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5m5b\" (UniqueName: \"kubernetes.io/projected/b22731b9-be09-4ad5-96ae-dce04203e0ad-kube-api-access-c5m5b\") pod \"node-ca-zc4j9\" (UID: \"b22731b9-be09-4ad5-96ae-dce04203e0ad\") " pod="openshift-image-registry/node-ca-zc4j9" Jul 07 14:24:38.557121 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556919 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-cnibin\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.557121 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556943 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-os-release\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.557121 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556967 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/68513391-0bac-4ef4-bf7a-2488aea5b76c-cni-binary-copy\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.557121 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.556992 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-etc-kubernetes\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.557121 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557018 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-os-release\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.557121 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557044 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.557121 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557070 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-hostroot\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.557121 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557114 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-system-cni-dir\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.557953 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557152 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm5g2\" (UniqueName: \"kubernetes.io/projected/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-kube-api-access-qm5g2\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.557953 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557181 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-slash\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.557953 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557216 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-run-systemd\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.557953 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557243 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-node-log\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.557953 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557266 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-log-socket\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.557953 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557291 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b22731b9-be09-4ad5-96ae-dce04203e0ad-serviceca\") pod \"node-ca-zc4j9\" (UID: \"b22731b9-be09-4ad5-96ae-dce04203e0ad\") " pod="openshift-image-registry/node-ca-zc4j9" Jul 07 14:24:38.557953 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557314 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-multus-cni-dir\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.557953 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557342 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-multus-socket-dir-parent\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.557953 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557368 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-host-var-lib-cni-bin\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.557953 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557424 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.557953 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557447 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-cni-bin\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.557953 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv588\" (UniqueName: \"kubernetes.io/projected/48af10ba-c8eb-42d6-a322-440ed4e09d81-kube-api-access-wv588\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.557953 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557517 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj498\" (UniqueName: \"kubernetes.io/projected/64b3ea37-09bc-491f-8e4c-de9cc97e7b5a-kube-api-access-wj498\") pod \"node-resolver-8xs5n\" (UID: \"64b3ea37-09bc-491f-8e4c-de9cc97e7b5a\") " pod="openshift-dns/node-resolver-8xs5n" Jul 07 14:24:38.557953 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557541 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-system-cni-dir\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.557953 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557596 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-host-var-lib-kubelet\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.557953 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557659 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-host-run-multus-certs\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.557953 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.557692 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6dtg\" (UniqueName: \"kubernetes.io/projected/68513391-0bac-4ef4-bf7a-2488aea5b76c-kube-api-access-b6dtg\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.558786 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.558014 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 07 14:24:38.558786 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.558606 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:38.558786 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:38.558676 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:24:38.560856 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.560830 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-np46r\"" Jul 07 14:24:38.561544 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.561042 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:24:38.561544 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.561280 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zx7p5\"" Jul 07 14:24:38.561544 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.561325 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 07 14:24:38.561544 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.561371 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 07 14:24:38.561544 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.561547 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 07 14:24:38.569518 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.569204 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 07 14:24:38.588824 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.588795 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vf2br" Jul 07 14:24:38.599681 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.599657 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vf2br" Jul 07 14:24:38.640824 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.640738 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 07 14:24:38.658180 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658132 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-sysconfig\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.658180 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658181 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjthj\" (UniqueName: \"kubernetes.io/projected/74893d67-336e-4e2b-9782-8f0e68db8463-kube-api-access-gjthj\") pod \"network-metrics-daemon-4bgbc\" (UID: \"74893d67-336e-4e2b-9782-8f0e68db8463\") " pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:38.658456 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658218 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-host-var-lib-cni-multus\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.658456 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658250 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.658456 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658306 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-var-lib-kubelet\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.658456 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658318 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-host-var-lib-cni-multus\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.658456 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658351 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret\") pod \"global-pull-secret-syncer-7w875\" (UID: \"fdf0d65e-e82f-4183-90df-fde1f0ce3435\") " pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:38.658456 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658402 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-host-run-netns\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.658456 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658432 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/48af10ba-c8eb-42d6-a322-440ed4e09d81-ovn-node-metrics-cert\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.658456 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658459 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-sysctl-d\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.658909 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658464 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-host-run-netns\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.658909 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658483 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-lib-modules\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.658909 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658483 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.658909 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658620 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-cnibin\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.658909 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658651 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-os-release\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.658909 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658689 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/68513391-0bac-4ef4-bf7a-2488aea5b76c-cni-binary-copy\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.658909 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658718 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.658909 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658746 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm5g2\" (UniqueName: \"kubernetes.io/projected/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-kube-api-access-qm5g2\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.658909 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658776 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-run-systemd\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.658909 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658804 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-sysctl-conf\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.658909 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658832 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-hostroot\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.658909 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658867 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-system-cni-dir\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.658909 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658905 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-node-log\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.659586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658932 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-log-socket\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.659586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658960 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-cni-bin\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.659586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658993 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv588\" (UniqueName: \"kubernetes.io/projected/48af10ba-c8eb-42d6-a322-440ed4e09d81-kube-api-access-wv588\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.659586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659025 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-systemd\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.659586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659053 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.659586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659078 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-kubernetes\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.659586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659104 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-host\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.659586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659128 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/431a400a-c4bd-4dc1-9a6c-db75e8a4e773-host-slash\") pod \"iptables-alerter-qch8s\" (UID: \"431a400a-c4bd-4dc1-9a6c-db75e8a4e773\") " pod="openshift-network-operator/iptables-alerter-qch8s" Jul 07 14:24:38.659586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659156 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.659586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659185 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj498\" (UniqueName: \"kubernetes.io/projected/64b3ea37-09bc-491f-8e4c-de9cc97e7b5a-kube-api-access-wj498\") pod \"node-resolver-8xs5n\" (UID: \"64b3ea37-09bc-491f-8e4c-de9cc97e7b5a\") " pod="openshift-dns/node-resolver-8xs5n" Jul 07 14:24:38.659586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659212 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-system-cni-dir\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.659586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659240 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-host-var-lib-kubelet\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.659586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659266 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-host-run-multus-certs\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.659586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659292 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-var-lib-openvswitch\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.659586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659293 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-cnibin\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.659586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659316 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-run\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.659586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659353 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fdf0d65e-e82f-4183-90df-fde1f0ce3435-kubelet-config\") pod \"global-pull-secret-syncer-7w875\" (UID: \"fdf0d65e-e82f-4183-90df-fde1f0ce3435\") " pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:38.660413 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659367 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-os-release\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.660413 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659380 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd5ws\" (UniqueName: \"kubernetes.io/projected/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-kube-api-access-dd5ws\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.660413 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659471 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-cnibin\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.660413 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659501 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-systemd-units\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.660413 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659528 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-run-netns\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.660413 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659560 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-etc-openvswitch\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.660413 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659588 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-run-openvswitch\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.660413 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659613 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-run-ovn-kubernetes\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.660413 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659643 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.660413 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659674 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkk7\" (UniqueName: \"kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7\") pod \"network-check-target-6b55m\" (UID: \"337cda64-fcd6-481c-86ed-b25d2396aaca\") " pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:24:38.660413 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659709 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-host-run-k8s-cni-cncf-io\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.660413 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659744 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/68513391-0bac-4ef4-bf7a-2488aea5b76c-multus-daemon-config\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.660413 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659771 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3de25b8b-594f-4628-bd9f-7dac25a945e6-tmp\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.660413 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659798 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/431a400a-c4bd-4dc1-9a6c-db75e8a4e773-iptables-alerter-script\") pod \"iptables-alerter-qch8s\" (UID: \"431a400a-c4bd-4dc1-9a6c-db75e8a4e773\") " pod="openshift-network-operator/iptables-alerter-qch8s" Jul 07 14:24:38.660413 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659826 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-device-dir\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.660413 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659853 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64b3ea37-09bc-491f-8e4c-de9cc97e7b5a-hosts-file\") pod \"node-resolver-8xs5n\" (UID: \"64b3ea37-09bc-491f-8e4c-de9cc97e7b5a\") " pod="openshift-dns/node-resolver-8xs5n" Jul 07 14:24:38.660413 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659879 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64b3ea37-09bc-491f-8e4c-de9cc97e7b5a-tmp-dir\") pod \"node-resolver-8xs5n\" (UID: \"64b3ea37-09bc-491f-8e4c-de9cc97e7b5a\") " pod="openshift-dns/node-resolver-8xs5n" Jul 07 14:24:38.661268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659906 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-cni-netd\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.661268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659935 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a09e6909-6763-46df-9d1c-1a920a322d59-agent-certs\") pod \"konnectivity-agent-88g99\" (UID: \"a09e6909-6763-46df-9d1c-1a920a322d59\") " pod="kube-system/konnectivity-agent-88g99" Jul 07 14:24:38.661268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659966 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.661268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659983 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/68513391-0bac-4ef4-bf7a-2488aea5b76c-cni-binary-copy\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.661268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.659994 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knsl7\" (UniqueName: \"kubernetes.io/projected/3de25b8b-594f-4628-bd9f-7dac25a945e6-kube-api-access-knsl7\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.661268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660026 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5m5b\" (UniqueName: \"kubernetes.io/projected/b22731b9-be09-4ad5-96ae-dce04203e0ad-kube-api-access-c5m5b\") pod \"node-ca-zc4j9\" (UID: \"b22731b9-be09-4ad5-96ae-dce04203e0ad\") " pod="openshift-image-registry/node-ca-zc4j9" Jul 07 14:24:38.661268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660054 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-etc-kubernetes\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.661268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660082 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-os-release\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.661268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660108 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/48af10ba-c8eb-42d6-a322-440ed4e09d81-ovnkube-config\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.661268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660135 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-modprobe-d\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.661268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660166 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fdf0d65e-e82f-4183-90df-fde1f0ce3435-dbus\") pod \"global-pull-secret-syncer-7w875\" (UID: \"fdf0d65e-e82f-4183-90df-fde1f0ce3435\") " pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:38.661268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660193 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-slash\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.661268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660220 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-etc-selinux\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.661268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660249 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-sys-fs\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.661268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660276 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b22731b9-be09-4ad5-96ae-dce04203e0ad-serviceca\") pod \"node-ca-zc4j9\" (UID: \"b22731b9-be09-4ad5-96ae-dce04203e0ad\") " pod="openshift-image-registry/node-ca-zc4j9" Jul 07 14:24:38.661268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660303 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-multus-cni-dir\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.661268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660330 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-multus-socket-dir-parent\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.662135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660359 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-host-var-lib-cni-bin\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.662135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660415 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-registration-dir\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.662135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660467 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6dtg\" (UniqueName: \"kubernetes.io/projected/68513391-0bac-4ef4-bf7a-2488aea5b76c-kube-api-access-b6dtg\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.662135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660498 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-kubelet\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.662135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660526 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-run-ovn\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.662135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660554 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/48af10ba-c8eb-42d6-a322-440ed4e09d81-env-overrides\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.662135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660560 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.662135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660581 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/48af10ba-c8eb-42d6-a322-440ed4e09d81-ovnkube-script-lib\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.662135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660608 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-sys\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.662135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660633 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.662135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660637 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs\") pod \"network-metrics-daemon-4bgbc\" (UID: \"74893d67-336e-4e2b-9782-8f0e68db8463\") " pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:38.662135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660669 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqr9h\" (UniqueName: \"kubernetes.io/projected/431a400a-c4bd-4dc1-9a6c-db75e8a4e773-kube-api-access-vqr9h\") pod \"iptables-alerter-qch8s\" (UID: \"431a400a-c4bd-4dc1-9a6c-db75e8a4e773\") " pod="openshift-network-operator/iptables-alerter-qch8s" Jul 07 14:24:38.662135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660692 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-hostroot\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.662135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660720 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b22731b9-be09-4ad5-96ae-dce04203e0ad-host\") pod \"node-ca-zc4j9\" (UID: \"b22731b9-be09-4ad5-96ae-dce04203e0ad\") " pod="openshift-image-registry/node-ca-zc4j9" Jul 07 14:24:38.662135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660734 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-system-cni-dir\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.662135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660749 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-multus-conf-dir\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.662135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660776 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-tuned\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.663004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660779 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-node-log\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.663004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660803 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-socket-dir\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.663004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660823 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-log-socket\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.663004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.658928 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 07 14:24:38.663004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.661132 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-host-run-k8s-cni-cncf-io\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.663004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.661220 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-multus-socket-dir-parent\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.663004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.661278 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-multus-cni-dir\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.663004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.661425 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-host-var-lib-cni-bin\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.663004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.661706 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-kubelet\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.663004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.661726 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b22731b9-be09-4ad5-96ae-dce04203e0ad-serviceca\") pod \"node-ca-zc4j9\" (UID: \"b22731b9-be09-4ad5-96ae-dce04203e0ad\") " pod="openshift-image-registry/node-ca-zc4j9" Jul 07 14:24:38.663004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.661758 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-run-ovn\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.663004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.662210 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/68513391-0bac-4ef4-bf7a-2488aea5b76c-multus-daemon-config\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.663004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.662232 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/48af10ba-c8eb-42d6-a322-440ed4e09d81-env-overrides\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.663004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.662314 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/64b3ea37-09bc-491f-8e4c-de9cc97e7b5a-hosts-file\") pod \"node-resolver-8xs5n\" (UID: \"64b3ea37-09bc-491f-8e4c-de9cc97e7b5a\") " pod="openshift-dns/node-resolver-8xs5n" Jul 07 14:24:38.663004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.662327 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-run-systemd\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.663004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.662384 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b22731b9-be09-4ad5-96ae-dce04203e0ad-host\") pod \"node-ca-zc4j9\" (UID: \"b22731b9-be09-4ad5-96ae-dce04203e0ad\") " pod="openshift-image-registry/node-ca-zc4j9" Jul 07 14:24:38.663004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.662433 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-multus-conf-dir\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.663004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.662536 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-host-run-multus-certs\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.664081 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.662580 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-cni-bin\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.664081 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.662615 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64b3ea37-09bc-491f-8e4c-de9cc97e7b5a-tmp-dir\") pod \"node-resolver-8xs5n\" (UID: \"64b3ea37-09bc-491f-8e4c-de9cc97e7b5a\") " pod="openshift-dns/node-resolver-8xs5n" Jul 07 14:24:38.664081 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.662855 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-run-openvswitch\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.664081 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.662867 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-cni-netd\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.664081 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.662909 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-run-netns\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.664081 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.662913 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-etc-openvswitch\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.664081 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.663137 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-etc-kubernetes\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.664081 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.663201 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-os-release\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.664081 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.663363 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.664081 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.663708 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/48af10ba-c8eb-42d6-a322-440ed4e09d81-ovnkube-config\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.664081 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.663749 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-run-ovn-kubernetes\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.664081 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.660832 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a09e6909-6763-46df-9d1c-1a920a322d59-konnectivity-ca\") pod \"konnectivity-agent-88g99\" (UID: \"a09e6909-6763-46df-9d1c-1a920a322d59\") " pod="kube-system/konnectivity-agent-88g99" Jul 07 14:24:38.664081 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.663783 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-host-slash\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.664081 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.663849 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-var-lib-openvswitch\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.664925 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.664310 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/48af10ba-c8eb-42d6-a322-440ed4e09d81-ovnkube-script-lib\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.664925 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.664364 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-cni-binary-copy\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.664925 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.664374 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-cnibin\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.664925 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.664421 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-host-var-lib-kubelet\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.664925 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.664433 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/68513391-0bac-4ef4-bf7a-2488aea5b76c-system-cni-dir\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.664925 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.664478 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/48af10ba-c8eb-42d6-a322-440ed4e09d81-systemd-units\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.665419 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.665396 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/48af10ba-c8eb-42d6-a322-440ed4e09d81-ovn-node-metrics-cert\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.676661 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:38.676293 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:24:38.676661 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:38.676323 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:24:38.676661 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:38.676339 2562 projected.go:196] Error preparing data for projected volume kube-api-access-xvkk7 for pod openshift-network-diagnostics/network-check-target-6b55m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:24:38.676661 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:38.676402 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7 podName:337cda64-fcd6-481c-86ed-b25d2396aaca nodeName:}" failed. No retries permitted until 2026-07-07 14:24:39.176379993 +0000 UTC m=+2.985927465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xvkk7" (UniqueName: "kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7") pod "network-check-target-6b55m" (UID: "337cda64-fcd6-481c-86ed-b25d2396aaca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:24:38.678279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.678210 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj498\" (UniqueName: \"kubernetes.io/projected/64b3ea37-09bc-491f-8e4c-de9cc97e7b5a-kube-api-access-wj498\") pod \"node-resolver-8xs5n\" (UID: \"64b3ea37-09bc-491f-8e4c-de9cc97e7b5a\") " pod="openshift-dns/node-resolver-8xs5n" Jul 07 14:24:38.682690 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.682638 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6dtg\" (UniqueName: \"kubernetes.io/projected/68513391-0bac-4ef4-bf7a-2488aea5b76c-kube-api-access-b6dtg\") pod \"multus-5gg2b\" (UID: \"68513391-0bac-4ef4-bf7a-2488aea5b76c\") " pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.683276 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.683249 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv588\" (UniqueName: \"kubernetes.io/projected/48af10ba-c8eb-42d6-a322-440ed4e09d81-kube-api-access-wv588\") pod \"ovnkube-node-zzfcn\" (UID: \"48af10ba-c8eb-42d6-a322-440ed4e09d81\") " pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.683842 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.683798 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5m5b\" (UniqueName: \"kubernetes.io/projected/b22731b9-be09-4ad5-96ae-dce04203e0ad-kube-api-access-c5m5b\") pod \"node-ca-zc4j9\" (UID: \"b22731b9-be09-4ad5-96ae-dce04203e0ad\") " pod="openshift-image-registry/node-ca-zc4j9" Jul 07 14:24:38.685598 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.685568 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm5g2\" (UniqueName: \"kubernetes.io/projected/8db8a8fe-45de-4a15-93f4-bd9d3c2019c7-kube-api-access-qm5g2\") pod \"multus-additional-cni-plugins-d74p5\" (UID: \"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7\") " pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.690799 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.690777 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 07 14:24:38.706235 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.706181 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.ec2.internal" event={"ID":"2b2a01d40505439c955139cb876044b6","Type":"ContainerStarted","Data":"d06ef322cc2c1547d1529bc0613634db18d63fb9b8fb5755697fb94d9a6efcfc"} Jul 07 14:24:38.708025 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.707963 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal" event={"ID":"4f5c9d3f2e625fd40d222604d069d6f5","Type":"ContainerStarted","Data":"e09b81106106144cd853f6a795ee769f1de7d40f19d76cf99822446f8df94623"} Jul 07 14:24:38.765293 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.764819 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3de25b8b-594f-4628-bd9f-7dac25a945e6-tmp\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.765293 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.764863 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/431a400a-c4bd-4dc1-9a6c-db75e8a4e773-iptables-alerter-script\") pod \"iptables-alerter-qch8s\" (UID: \"431a400a-c4bd-4dc1-9a6c-db75e8a4e773\") " pod="openshift-network-operator/iptables-alerter-qch8s" Jul 07 14:24:38.765293 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.764891 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-device-dir\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.765293 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.764922 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a09e6909-6763-46df-9d1c-1a920a322d59-agent-certs\") pod \"konnectivity-agent-88g99\" (UID: \"a09e6909-6763-46df-9d1c-1a920a322d59\") " pod="kube-system/konnectivity-agent-88g99" Jul 07 14:24:38.765293 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.764952 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knsl7\" (UniqueName: \"kubernetes.io/projected/3de25b8b-594f-4628-bd9f-7dac25a945e6-kube-api-access-knsl7\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.765293 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.764986 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-modprobe-d\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.765293 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765011 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fdf0d65e-e82f-4183-90df-fde1f0ce3435-dbus\") pod \"global-pull-secret-syncer-7w875\" (UID: \"fdf0d65e-e82f-4183-90df-fde1f0ce3435\") " pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:38.765293 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765038 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-etc-selinux\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.765293 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765062 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-sys-fs\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.765293 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765091 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-registration-dir\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.765293 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765120 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-sys\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.765293 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765147 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs\") pod \"network-metrics-daemon-4bgbc\" (UID: \"74893d67-336e-4e2b-9782-8f0e68db8463\") " pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:38.765293 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqr9h\" (UniqueName: \"kubernetes.io/projected/431a400a-c4bd-4dc1-9a6c-db75e8a4e773-kube-api-access-vqr9h\") pod \"iptables-alerter-qch8s\" (UID: \"431a400a-c4bd-4dc1-9a6c-db75e8a4e773\") " pod="openshift-network-operator/iptables-alerter-qch8s" Jul 07 14:24:38.765293 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765230 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-tuned\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.765293 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765258 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-socket-dir\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.765293 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765283 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a09e6909-6763-46df-9d1c-1a920a322d59-konnectivity-ca\") pod \"konnectivity-agent-88g99\" (UID: \"a09e6909-6763-46df-9d1c-1a920a322d59\") " pod="kube-system/konnectivity-agent-88g99" Jul 07 14:24:38.765293 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765311 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-sysconfig\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.766344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765339 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjthj\" (UniqueName: \"kubernetes.io/projected/74893d67-336e-4e2b-9782-8f0e68db8463-kube-api-access-gjthj\") pod \"network-metrics-daemon-4bgbc\" (UID: \"74893d67-336e-4e2b-9782-8f0e68db8463\") " pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:38.766344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765367 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-var-lib-kubelet\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.766344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765395 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret\") pod \"global-pull-secret-syncer-7w875\" (UID: \"fdf0d65e-e82f-4183-90df-fde1f0ce3435\") " pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:38.766344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765425 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-sysctl-d\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.766344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765471 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-lib-modules\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.766344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765505 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-sysctl-conf\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.766344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765599 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-sys-fs\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.766344 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:38.765627 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:24:38.766344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765656 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-systemd\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.766344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765691 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-kubernetes\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.766344 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:38.765722 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs podName:74893d67-336e-4e2b-9782-8f0e68db8463 nodeName:}" failed. No retries permitted until 2026-07-07 14:24:39.265697384 +0000 UTC m=+3.075244879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs") pod "network-metrics-daemon-4bgbc" (UID: "74893d67-336e-4e2b-9782-8f0e68db8463") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:24:38.766344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765739 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-kubernetes\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.766344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765756 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-modprobe-d\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.766344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765758 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-host\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.766344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765796 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-registration-dir\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.766344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765804 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-host\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.766344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765824 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/431a400a-c4bd-4dc1-9a6c-db75e8a4e773-host-slash\") pod \"iptables-alerter-qch8s\" (UID: \"431a400a-c4bd-4dc1-9a6c-db75e8a4e773\") " pod="openshift-network-operator/iptables-alerter-qch8s" Jul 07 14:24:38.767242 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765848 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-sys\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.767242 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765880 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.767242 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765919 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-run\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.767242 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765949 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fdf0d65e-e82f-4183-90df-fde1f0ce3435-kubelet-config\") pod \"global-pull-secret-syncer-7w875\" (UID: \"fdf0d65e-e82f-4183-90df-fde1f0ce3435\") " pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:38.767242 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.765987 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-sysctl-conf\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.767242 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.766007 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd5ws\" (UniqueName: \"kubernetes.io/projected/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-kube-api-access-dd5ws\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.767242 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.766087 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-etc-selinux\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.767242 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.766134 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-device-dir\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.767242 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.766184 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-systemd\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.767242 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.766336 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.767242 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.766403 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-run\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.767242 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.766471 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fdf0d65e-e82f-4183-90df-fde1f0ce3435-kubelet-config\") pod \"global-pull-secret-syncer-7w875\" (UID: \"fdf0d65e-e82f-4183-90df-fde1f0ce3435\") " pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:38.767242 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.766772 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/431a400a-c4bd-4dc1-9a6c-db75e8a4e773-iptables-alerter-script\") pod \"iptables-alerter-qch8s\" (UID: \"431a400a-c4bd-4dc1-9a6c-db75e8a4e773\") " pod="openshift-network-operator/iptables-alerter-qch8s" Jul 07 14:24:38.767242 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.766800 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/431a400a-c4bd-4dc1-9a6c-db75e8a4e773-host-slash\") pod \"iptables-alerter-qch8s\" (UID: \"431a400a-c4bd-4dc1-9a6c-db75e8a4e773\") " pod="openshift-network-operator/iptables-alerter-qch8s" Jul 07 14:24:38.767242 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:38.766910 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:24:38.767242 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:38.766978 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret podName:fdf0d65e-e82f-4183-90df-fde1f0ce3435 nodeName:}" failed. No retries permitted until 2026-07-07 14:24:39.266959887 +0000 UTC m=+3.076507371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret") pod "global-pull-secret-syncer-7w875" (UID: "fdf0d65e-e82f-4183-90df-fde1f0ce3435") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:24:38.767242 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.766986 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-sysctl-d\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.768291 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.767039 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-var-lib-kubelet\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.768291 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.767107 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-socket-dir\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.768291 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.767176 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-sysconfig\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.768291 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.767179 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fdf0d65e-e82f-4183-90df-fde1f0ce3435-dbus\") pod \"global-pull-secret-syncer-7w875\" (UID: \"fdf0d65e-e82f-4183-90df-fde1f0ce3435\") " pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:38.768291 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.767340 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3de25b8b-594f-4628-bd9f-7dac25a945e6-lib-modules\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.768291 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.767812 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a09e6909-6763-46df-9d1c-1a920a322d59-konnectivity-ca\") pod \"konnectivity-agent-88g99\" (UID: \"a09e6909-6763-46df-9d1c-1a920a322d59\") " pod="kube-system/konnectivity-agent-88g99" Jul 07 14:24:38.770606 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.770548 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3de25b8b-594f-4628-bd9f-7dac25a945e6-tmp\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.770606 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.770559 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a09e6909-6763-46df-9d1c-1a920a322d59-agent-certs\") pod \"konnectivity-agent-88g99\" (UID: \"a09e6909-6763-46df-9d1c-1a920a322d59\") " pod="kube-system/konnectivity-agent-88g99" Jul 07 14:24:38.774182 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.774156 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3de25b8b-594f-4628-bd9f-7dac25a945e6-etc-tuned\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.790718 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.790682 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knsl7\" (UniqueName: \"kubernetes.io/projected/3de25b8b-594f-4628-bd9f-7dac25a945e6-kube-api-access-knsl7\") pod \"tuned-7frb5\" (UID: \"3de25b8b-594f-4628-bd9f-7dac25a945e6\") " pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.791195 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.791167 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqr9h\" (UniqueName: \"kubernetes.io/projected/431a400a-c4bd-4dc1-9a6c-db75e8a4e773-kube-api-access-vqr9h\") pod \"iptables-alerter-qch8s\" (UID: \"431a400a-c4bd-4dc1-9a6c-db75e8a4e773\") " pod="openshift-network-operator/iptables-alerter-qch8s" Jul 07 14:24:38.791666 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.791637 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd5ws\" (UniqueName: \"kubernetes.io/projected/0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41-kube-api-access-dd5ws\") pod \"aws-ebs-csi-driver-node-cvgk6\" (UID: \"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.792774 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.792744 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjthj\" (UniqueName: \"kubernetes.io/projected/74893d67-336e-4e2b-9782-8f0e68db8463-kube-api-access-gjthj\") pod \"network-metrics-daemon-4bgbc\" (UID: \"74893d67-336e-4e2b-9782-8f0e68db8463\") " pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:38.847683 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.847636 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:24:38.856337 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.856302 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8xs5n" Jul 07 14:24:38.858567 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:38.858537 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48af10ba_c8eb_42d6_a322_440ed4e09d81.slice/crio-048e42683b448ecbaddfaebb97452edc04950194259045d27b2fdc68b9db6ea6 WatchSource:0}: Error finding container 048e42683b448ecbaddfaebb97452edc04950194259045d27b2fdc68b9db6ea6: Status 404 returned error can't find the container with id 048e42683b448ecbaddfaebb97452edc04950194259045d27b2fdc68b9db6ea6 Jul 07 14:24:38.867334 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:38.867304 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64b3ea37_09bc_491f_8e4c_de9cc97e7b5a.slice/crio-58ecc07d7152234053e144b3e36c841457fb0fcc855efa1afb7269d564dfe32c WatchSource:0}: Error finding container 58ecc07d7152234053e144b3e36c841457fb0fcc855efa1afb7269d564dfe32c: Status 404 returned error can't find the container with id 58ecc07d7152234053e144b3e36c841457fb0fcc855efa1afb7269d564dfe32c Jul 07 14:24:38.869335 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.868550 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zc4j9" Jul 07 14:24:38.877775 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.877751 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5gg2b" Jul 07 14:24:38.880489 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:38.880361 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22731b9_be09_4ad5_96ae_dce04203e0ad.slice/crio-e83368f4f5c673d5f57714710def99bb7b443c72a038d50513269d17b362f879 WatchSource:0}: Error finding container e83368f4f5c673d5f57714710def99bb7b443c72a038d50513269d17b362f879: Status 404 returned error can't find the container with id e83368f4f5c673d5f57714710def99bb7b443c72a038d50513269d17b362f879 Jul 07 14:24:38.885790 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.885769 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d74p5" Jul 07 14:24:38.888624 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:38.888586 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68513391_0bac_4ef4_bf7a_2488aea5b76c.slice/crio-d5b2e3519549d32458dabfbc1d796683c64053aa643fb1169c3400b42ae5ebcc WatchSource:0}: Error finding container d5b2e3519549d32458dabfbc1d796683c64053aa643fb1169c3400b42ae5ebcc: Status 404 returned error can't find the container with id d5b2e3519549d32458dabfbc1d796683c64053aa643fb1169c3400b42ae5ebcc Jul 07 14:24:38.893969 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.893946 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-88g99" Jul 07 14:24:38.895016 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:38.894990 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db8a8fe_45de_4a15_93f4_bd9d3c2019c7.slice/crio-9c21b907a26a264006a29a23e57938ffb7374ff7ae2524878ce133cac4bdee6f WatchSource:0}: Error finding container 9c21b907a26a264006a29a23e57938ffb7374ff7ae2524878ce133cac4bdee6f: Status 404 returned error can't find the container with id 9c21b907a26a264006a29a23e57938ffb7374ff7ae2524878ce133cac4bdee6f Jul 07 14:24:38.903020 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:38.902995 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09e6909_6763_46df_9d1c_1a920a322d59.slice/crio-c85ce017c0cde47b8751d83705c332bb804cb20c07213c2abb07c9c1aa0afb1a WatchSource:0}: Error finding container c85ce017c0cde47b8751d83705c332bb804cb20c07213c2abb07c9c1aa0afb1a: Status 404 returned error can't find the container with id c85ce017c0cde47b8751d83705c332bb804cb20c07213c2abb07c9c1aa0afb1a Jul 07 14:24:38.907614 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.907591 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qch8s" Jul 07 14:24:38.916347 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.916322 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7frb5" Jul 07 14:24:38.918546 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:38.918515 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431a400a_c4bd_4dc1_9a6c_db75e8a4e773.slice/crio-0498cb2f233018697719b9c8f89c8d01944fe1aaca47eb9f1dd6e36df5ee9756 WatchSource:0}: Error finding container 0498cb2f233018697719b9c8f89c8d01944fe1aaca47eb9f1dd6e36df5ee9756: Status 404 returned error can't find the container with id 0498cb2f233018697719b9c8f89c8d01944fe1aaca47eb9f1dd6e36df5ee9756 Jul 07 14:24:38.924943 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:38.924920 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" Jul 07 14:24:38.925130 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:24:38.925100 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de25b8b_594f_4628_bd9f_7dac25a945e6.slice/crio-7a299b8fa07ec6a42f650d5f47313eaa82f483117bc7f78dbb939840345d227c WatchSource:0}: Error finding container 7a299b8fa07ec6a42f650d5f47313eaa82f483117bc7f78dbb939840345d227c: Status 404 returned error can't find the container with id 7a299b8fa07ec6a42f650d5f47313eaa82f483117bc7f78dbb939840345d227c Jul 07 14:24:39.270209 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:39.270111 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkk7\" (UniqueName: \"kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7\") pod \"network-check-target-6b55m\" (UID: \"337cda64-fcd6-481c-86ed-b25d2396aaca\") " pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:24:39.270209 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:39.270183 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs\") pod \"network-metrics-daemon-4bgbc\" (UID: \"74893d67-336e-4e2b-9782-8f0e68db8463\") " pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:39.270468 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:39.270224 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret\") pod \"global-pull-secret-syncer-7w875\" (UID: \"fdf0d65e-e82f-4183-90df-fde1f0ce3435\") " pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:39.270468 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:39.270296 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:24:39.270468 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:39.270307 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:24:39.270468 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:39.270332 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:24:39.270468 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:39.270345 2562 projected.go:196] Error preparing data for projected volume kube-api-access-xvkk7 for pod openshift-network-diagnostics/network-check-target-6b55m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:24:39.270468 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:39.270343 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:24:39.270468 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:39.270380 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs podName:74893d67-336e-4e2b-9782-8f0e68db8463 nodeName:}" failed. No retries permitted until 2026-07-07 14:24:40.27035984 +0000 UTC m=+4.079907325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs") pod "network-metrics-daemon-4bgbc" (UID: "74893d67-336e-4e2b-9782-8f0e68db8463") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:24:39.270468 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:39.270420 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7 podName:337cda64-fcd6-481c-86ed-b25d2396aaca nodeName:}" failed. No retries permitted until 2026-07-07 14:24:40.270408958 +0000 UTC m=+4.079956429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xvkk7" (UniqueName: "kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7") pod "network-check-target-6b55m" (UID: "337cda64-fcd6-481c-86ed-b25d2396aaca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:24:39.270468 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:39.270437 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret podName:fdf0d65e-e82f-4183-90df-fde1f0ce3435 nodeName:}" failed. No retries permitted until 2026-07-07 14:24:40.27042849 +0000 UTC m=+4.079975963 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret") pod "global-pull-secret-syncer-7w875" (UID: "fdf0d65e-e82f-4183-90df-fde1f0ce3435") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:24:39.602939 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:39.602811 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:19:38 +0000 UTC" deadline="2028-02-18 10:27:50.542308192 +0000 UTC" Jul 07 14:24:39.602939 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:39.602884 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14180h3m10.939428713s" Jul 07 14:24:39.701087 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:39.701016 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:24:39.701292 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:39.701152 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:24:39.701655 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:39.701629 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:39.701803 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:39.701745 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:24:39.716639 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:39.716533 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" event={"ID":"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41","Type":"ContainerStarted","Data":"44326654172f8f737d0dc0fefe876b2e42380bbf2e840b2915cac030efc5e748"} Jul 07 14:24:39.718580 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:39.718487 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7frb5" event={"ID":"3de25b8b-594f-4628-bd9f-7dac25a945e6","Type":"ContainerStarted","Data":"7a299b8fa07ec6a42f650d5f47313eaa82f483117bc7f78dbb939840345d227c"} Jul 07 14:24:39.720904 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:39.720813 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qch8s" event={"ID":"431a400a-c4bd-4dc1-9a6c-db75e8a4e773","Type":"ContainerStarted","Data":"0498cb2f233018697719b9c8f89c8d01944fe1aaca47eb9f1dd6e36df5ee9756"} Jul 07 14:24:39.723594 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:39.723568 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d74p5" event={"ID":"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7","Type":"ContainerStarted","Data":"9c21b907a26a264006a29a23e57938ffb7374ff7ae2524878ce133cac4bdee6f"} Jul 07 14:24:39.725708 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:39.725683 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8xs5n" event={"ID":"64b3ea37-09bc-491f-8e4c-de9cc97e7b5a","Type":"ContainerStarted","Data":"58ecc07d7152234053e144b3e36c841457fb0fcc855efa1afb7269d564dfe32c"} Jul 07 14:24:39.727464 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:39.727425 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-88g99" event={"ID":"a09e6909-6763-46df-9d1c-1a920a322d59","Type":"ContainerStarted","Data":"c85ce017c0cde47b8751d83705c332bb804cb20c07213c2abb07c9c1aa0afb1a"} Jul 07 14:24:39.730354 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:39.730326 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5gg2b" event={"ID":"68513391-0bac-4ef4-bf7a-2488aea5b76c","Type":"ContainerStarted","Data":"d5b2e3519549d32458dabfbc1d796683c64053aa643fb1169c3400b42ae5ebcc"} Jul 07 14:24:39.732037 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:39.732002 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zc4j9" event={"ID":"b22731b9-be09-4ad5-96ae-dce04203e0ad","Type":"ContainerStarted","Data":"e83368f4f5c673d5f57714710def99bb7b443c72a038d50513269d17b362f879"} Jul 07 14:24:39.733685 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:39.733636 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" event={"ID":"48af10ba-c8eb-42d6-a322-440ed4e09d81","Type":"ContainerStarted","Data":"048e42683b448ecbaddfaebb97452edc04950194259045d27b2fdc68b9db6ea6"} Jul 07 14:24:40.279198 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:40.279151 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs\") pod \"network-metrics-daemon-4bgbc\" (UID: \"74893d67-336e-4e2b-9782-8f0e68db8463\") " pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:40.279404 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:40.279231 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret\") pod \"global-pull-secret-syncer-7w875\" (UID: \"fdf0d65e-e82f-4183-90df-fde1f0ce3435\") " pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:40.279404 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:40.279290 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkk7\" (UniqueName: \"kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7\") pod \"network-check-target-6b55m\" (UID: \"337cda64-fcd6-481c-86ed-b25d2396aaca\") " pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:24:40.279573 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:40.279467 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:24:40.279573 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:40.279486 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:24:40.279573 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:40.279499 2562 projected.go:196] Error preparing data for projected volume kube-api-access-xvkk7 for pod openshift-network-diagnostics/network-check-target-6b55m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:24:40.279732 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:40.279575 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7 podName:337cda64-fcd6-481c-86ed-b25d2396aaca nodeName:}" failed. No retries permitted until 2026-07-07 14:24:42.279544454 +0000 UTC m=+6.089091926 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xvkk7" (UniqueName: "kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7") pod "network-check-target-6b55m" (UID: "337cda64-fcd6-481c-86ed-b25d2396aaca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:24:40.280055 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:40.280028 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:24:40.280174 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:40.280102 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs podName:74893d67-336e-4e2b-9782-8f0e68db8463 nodeName:}" failed. No retries permitted until 2026-07-07 14:24:42.280075259 +0000 UTC m=+6.089622736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs") pod "network-metrics-daemon-4bgbc" (UID: "74893d67-336e-4e2b-9782-8f0e68db8463") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:24:40.280174 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:40.280165 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:24:40.280296 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:40.280199 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret podName:fdf0d65e-e82f-4183-90df-fde1f0ce3435 nodeName:}" failed. No retries permitted until 2026-07-07 14:24:42.280187555 +0000 UTC m=+6.089735027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret") pod "global-pull-secret-syncer-7w875" (UID: "fdf0d65e-e82f-4183-90df-fde1f0ce3435") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:24:40.603673 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:40.603591 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 14:19:38 +0000 UTC" deadline="2028-04-18 07:22:52.974007682 +0000 UTC" Jul 07 14:24:40.603673 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:40.603658 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15616h58m12.370354223s" Jul 07 14:24:40.703795 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:40.703762 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:40.704051 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:40.703920 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:24:41.701537 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:41.701504 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:24:41.702066 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:41.701643 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:24:41.702134 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:41.702081 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:41.702242 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:41.702185 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:24:42.300832 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:42.300425 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs\") pod \"network-metrics-daemon-4bgbc\" (UID: \"74893d67-336e-4e2b-9782-8f0e68db8463\") " pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:42.300832 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:42.300485 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret\") pod \"global-pull-secret-syncer-7w875\" (UID: \"fdf0d65e-e82f-4183-90df-fde1f0ce3435\") " pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:42.300832 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:42.300546 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkk7\" (UniqueName: \"kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7\") pod \"network-check-target-6b55m\" (UID: \"337cda64-fcd6-481c-86ed-b25d2396aaca\") " pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:24:42.300832 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:42.300598 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:24:42.300832 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:42.300679 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs podName:74893d67-336e-4e2b-9782-8f0e68db8463 nodeName:}" failed. No retries permitted until 2026-07-07 14:24:46.300657799 +0000 UTC m=+10.110205269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs") pod "network-metrics-daemon-4bgbc" (UID: "74893d67-336e-4e2b-9782-8f0e68db8463") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:24:42.300832 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:42.300681 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:24:42.300832 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:42.300721 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:24:42.300832 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:42.300739 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:24:42.300832 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:42.300747 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret podName:fdf0d65e-e82f-4183-90df-fde1f0ce3435 nodeName:}" failed. No retries permitted until 2026-07-07 14:24:46.300729408 +0000 UTC m=+10.110276892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret") pod "global-pull-secret-syncer-7w875" (UID: "fdf0d65e-e82f-4183-90df-fde1f0ce3435") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:24:42.300832 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:42.300752 2562 projected.go:196] Error preparing data for projected volume kube-api-access-xvkk7 for pod openshift-network-diagnostics/network-check-target-6b55m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:24:42.300832 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:42.300804 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7 podName:337cda64-fcd6-481c-86ed-b25d2396aaca nodeName:}" failed. No retries permitted until 2026-07-07 14:24:46.30078892 +0000 UTC m=+10.110336391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xvkk7" (UniqueName: "kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7") pod "network-check-target-6b55m" (UID: "337cda64-fcd6-481c-86ed-b25d2396aaca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:24:42.701730 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:42.701645 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:42.702219 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:42.701808 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:24:43.701742 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:43.701340 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:43.701742 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:43.701499 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:24:43.701742 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:43.701579 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:24:43.701742 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:43.701725 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:24:44.701447 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:44.701193 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:44.701584 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:44.701485 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:24:44.757866 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:44.757827 2562 generic.go:356] "Generic (PLEG): container finished" podID="4f5c9d3f2e625fd40d222604d069d6f5" containerID="4a638c7e8ee74d2860c0f662e311cdca6eef92d551c40b55571c6f003d864128" exitCode=0 Jul 07 14:24:44.758322 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:44.757885 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal" event={"ID":"4f5c9d3f2e625fd40d222604d069d6f5","Type":"ContainerDied","Data":"4a638c7e8ee74d2860c0f662e311cdca6eef92d551c40b55571c6f003d864128"} Jul 07 14:24:45.700859 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:45.700523 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:24:45.700859 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:45.700667 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:24:45.701111 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:45.701045 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:45.701169 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:45.701139 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:24:46.335267 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:46.335224 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkk7\" (UniqueName: \"kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7\") pod \"network-check-target-6b55m\" (UID: \"337cda64-fcd6-481c-86ed-b25d2396aaca\") " pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:24:46.335748 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:46.335298 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs\") pod \"network-metrics-daemon-4bgbc\" (UID: \"74893d67-336e-4e2b-9782-8f0e68db8463\") " pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:46.335748 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:46.335334 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret\") pod \"global-pull-secret-syncer-7w875\" (UID: \"fdf0d65e-e82f-4183-90df-fde1f0ce3435\") " pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:46.335748 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:46.335418 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:24:46.335748 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:46.335435 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:24:46.335748 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:46.335444 2562 projected.go:196] Error preparing data for projected volume kube-api-access-xvkk7 for pod openshift-network-diagnostics/network-check-target-6b55m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:24:46.335748 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:46.335483 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:24:46.335748 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:46.335507 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7 podName:337cda64-fcd6-481c-86ed-b25d2396aaca nodeName:}" failed. No retries permitted until 2026-07-07 14:24:54.335488944 +0000 UTC m=+18.145036417 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xvkk7" (UniqueName: "kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7") pod "network-check-target-6b55m" (UID: "337cda64-fcd6-481c-86ed-b25d2396aaca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:24:46.335748 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:46.335524 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:24:46.335748 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:46.335549 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs podName:74893d67-336e-4e2b-9782-8f0e68db8463 nodeName:}" failed. No retries permitted until 2026-07-07 14:24:54.33553049 +0000 UTC m=+18.145077961 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs") pod "network-metrics-daemon-4bgbc" (UID: "74893d67-336e-4e2b-9782-8f0e68db8463") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:24:46.335748 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:46.335571 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret podName:fdf0d65e-e82f-4183-90df-fde1f0ce3435 nodeName:}" failed. No retries permitted until 2026-07-07 14:24:54.335559593 +0000 UTC m=+18.145107068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret") pod "global-pull-secret-syncer-7w875" (UID: "fdf0d65e-e82f-4183-90df-fde1f0ce3435") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:24:46.702325 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:46.702231 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:46.702539 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:46.702363 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:24:47.700925 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:47.700840 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:24:47.701361 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:47.700973 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:24:47.701361 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:47.701026 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:47.701361 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:47.701143 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:24:48.700909 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:48.700723 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:48.700909 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:48.700867 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:24:49.701141 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:49.701097 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:49.701141 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:49.701135 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:24:49.701630 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:49.701271 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:24:49.701630 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:49.701438 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:24:50.700736 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:50.700691 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:50.700921 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:50.700829 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:24:51.701264 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:51.701227 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:51.701264 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:51.701254 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:24:51.701781 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:51.701382 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:24:51.701781 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:51.701489 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:24:52.701652 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:52.701615 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:52.702096 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:52.701765 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:24:53.700850 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:53.700820 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:24:53.701023 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:53.700911 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:24:53.701023 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:53.700964 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:53.701095 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:53.701046 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:24:54.399429 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:54.399387 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs\") pod \"network-metrics-daemon-4bgbc\" (UID: \"74893d67-336e-4e2b-9782-8f0e68db8463\") " pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:54.399918 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:54.399442 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret\") pod \"global-pull-secret-syncer-7w875\" (UID: \"fdf0d65e-e82f-4183-90df-fde1f0ce3435\") " pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:54.399918 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:54.399494 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkk7\" (UniqueName: \"kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7\") pod \"network-check-target-6b55m\" (UID: \"337cda64-fcd6-481c-86ed-b25d2396aaca\") " pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:24:54.399918 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:54.399569 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:24:54.399918 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:54.399596 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:24:54.399918 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:54.399614 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:24:54.399918 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:54.399627 2562 projected.go:196] Error preparing data for projected volume kube-api-access-xvkk7 for pod openshift-network-diagnostics/network-check-target-6b55m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:24:54.399918 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:54.399642 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs podName:74893d67-336e-4e2b-9782-8f0e68db8463 nodeName:}" failed. No retries permitted until 2026-07-07 14:25:10.39962106 +0000 UTC m=+34.209168540 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs") pod "network-metrics-daemon-4bgbc" (UID: "74893d67-336e-4e2b-9782-8f0e68db8463") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:24:54.399918 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:54.399579 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:24:54.399918 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:54.399674 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7 podName:337cda64-fcd6-481c-86ed-b25d2396aaca nodeName:}" failed. No retries permitted until 2026-07-07 14:25:10.399658888 +0000 UTC m=+34.209206357 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xvkk7" (UniqueName: "kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7") pod "network-check-target-6b55m" (UID: "337cda64-fcd6-481c-86ed-b25d2396aaca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:24:54.399918 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:54.399703 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret podName:fdf0d65e-e82f-4183-90df-fde1f0ce3435 nodeName:}" failed. No retries permitted until 2026-07-07 14:25:10.399691037 +0000 UTC m=+34.209238508 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret") pod "global-pull-secret-syncer-7w875" (UID: "fdf0d65e-e82f-4183-90df-fde1f0ce3435") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:24:54.701105 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:54.701016 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:54.701288 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:54.701175 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:24:55.701505 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:55.701469 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:24:55.701505 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:55.701492 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:55.701947 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:55.701598 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:24:55.701947 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:55.701730 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:24:56.703392 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:56.703338 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:56.703825 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:56.703678 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:24:57.701163 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:57.701125 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:24:57.701355 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:57.701125 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:57.701355 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:57.701258 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:24:57.701355 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:57.701331 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:24:58.701135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.700852 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:24:58.701135 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:58.701034 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:24:58.785249 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.784943 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" event={"ID":"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41","Type":"ContainerStarted","Data":"049da0eda3df270c69bfd51394229afe6f58d644700536b57d10cdc44aae65d1"} Jul 07 14:24:58.786572 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.786544 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7frb5" event={"ID":"3de25b8b-594f-4628-bd9f-7dac25a945e6","Type":"ContainerStarted","Data":"0ff51f1cb47e02484acfa5450e9faec9642e4ad2d07534a8c4501f5c29fdc8cb"} Jul 07 14:24:58.788316 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.788295 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d74p5" event={"ID":"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7","Type":"ContainerStarted","Data":"cf7d313305902bd21f2ca034b5fb79ecb3c9bff2fa36888fe7a0d39432366449"} Jul 07 14:24:58.790709 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.790683 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8xs5n" event={"ID":"64b3ea37-09bc-491f-8e4c-de9cc97e7b5a","Type":"ContainerStarted","Data":"d55bc7cc1d43c5c95997dee71e5e52809561ade72f57624707037afac0dee291"} Jul 07 14:24:58.792687 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.792659 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.ec2.internal" event={"ID":"2b2a01d40505439c955139cb876044b6","Type":"ContainerStarted","Data":"c9aa858ba60c2f75d4d82fc5c56eed349130abbb8e00668bddbe786b5e28df39"} Jul 07 14:24:58.794725 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.794699 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal" event={"ID":"4f5c9d3f2e625fd40d222604d069d6f5","Type":"ContainerStarted","Data":"79d308a20519ac12486ec8742208bd725e1183ad27a5f30109128d41133ed0c0"} Jul 07 14:24:58.796205 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.796181 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-88g99" event={"ID":"a09e6909-6763-46df-9d1c-1a920a322d59","Type":"ContainerStarted","Data":"921335446ed86ea2a996418242887d531b6b03699a65c8ac56be3f6c8e0d2ec2"} Jul 07 14:24:58.797684 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.797655 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5gg2b" event={"ID":"68513391-0bac-4ef4-bf7a-2488aea5b76c","Type":"ContainerStarted","Data":"d4620a52af0ca5daa26302a0052a1e4c4acbccb48fee9aaca842ac9cfa6b78bd"} Jul 07 14:24:58.802388 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.802276 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zc4j9" event={"ID":"b22731b9-be09-4ad5-96ae-dce04203e0ad","Type":"ContainerStarted","Data":"4219c4c6c64e5243611249a652c7e15699d6ae6b2a46d0f28c14e727113eb181"} Jul 07 14:24:58.804093 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.804073 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" event={"ID":"48af10ba-c8eb-42d6-a322-440ed4e09d81","Type":"ContainerStarted","Data":"d2978c6c6a7c235a4562019a2acf5a5efd9abb930e2c189ed193ec65f92c219e"} Jul 07 14:24:58.804180 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.804145 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" event={"ID":"48af10ba-c8eb-42d6-a322-440ed4e09d81","Type":"ContainerStarted","Data":"bb2dfc607ec7c05ddccdfe948f2af3e06c0622afd845cb4d79a74ee6b187e9ea"} Jul 07 14:24:58.804709 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.804661 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7frb5" podStartSLOduration=2.3158752160000002 podStartE2EDuration="21.804650472s" totalImagesPullingTime="19.488775256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:24:58.80461681 +0000 UTC m=+22.614164311" watchObservedRunningTime="2026-07-07 14:24:58.804650472 +0000 UTC m=+22.614197962" Jul 07 14:24:58.822467 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.822415 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-34.ec2.internal" podStartSLOduration=20.822394573 podStartE2EDuration="20.822394573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:24:58.822332664 +0000 UTC m=+22.631880154" watchObservedRunningTime="2026-07-07 14:24:58.822394573 +0000 UTC m=+22.631942060" Jul 07 14:24:58.839077 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.839028 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8xs5n" podStartSLOduration=2.295218379 podStartE2EDuration="21.839014208s" totalImagesPullingTime="19.543795829s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:24:58.838713447 +0000 UTC m=+22.648260936" watchObservedRunningTime="2026-07-07 14:24:58.839014208 +0000 UTC m=+22.648561698" Jul 07 14:24:58.854845 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.854798 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zc4j9" podStartSLOduration=2.342911804 podStartE2EDuration="21.854782831s" totalImagesPullingTime="19.511871027s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:24:58.854742644 +0000 UTC m=+22.664290146" watchObservedRunningTime="2026-07-07 14:24:58.854782831 +0000 UTC m=+22.664330321" Jul 07 14:24:58.876906 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.876857 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5gg2b" podStartSLOduration=2.283319308 podStartE2EDuration="21.876841379s" totalImagesPullingTime="19.593522071s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:24:58.876741126 +0000 UTC m=+22.686288616" watchObservedRunningTime="2026-07-07 14:24:58.876841379 +0000 UTC m=+22.686388866" Jul 07 14:24:58.923877 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.923832 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-88g99" podStartSLOduration=2.486428617 podStartE2EDuration="21.923816891s" totalImagesPullingTime="19.437388274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:24:58.923388898 +0000 UTC m=+22.732936387" watchObservedRunningTime="2026-07-07 14:24:58.923816891 +0000 UTC m=+22.733364394" Jul 07 14:24:58.940349 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.940275 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-34.ec2.internal" podStartSLOduration=20.940259727 podStartE2EDuration="20.940259727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:24:58.939852524 +0000 UTC m=+22.749400013" watchObservedRunningTime="2026-07-07 14:24:58.940259727 +0000 UTC m=+22.749807216" Jul 07 14:24:58.979937 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:58.979901 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:24:59.077984 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:59.077936 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:24:59.179798 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:59.179732 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:24:59.701128 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:59.701096 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:24:59.701340 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:59.701098 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:24:59.701340 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:59.701216 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:24:59.701340 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:24:59.701303 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:24:59.776898 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:59.776862 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:24:59.809458 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:59.809424 2562 generic.go:356] "Generic (PLEG): container finished" podID="8db8a8fe-45de-4a15-93f4-bd9d3c2019c7" containerID="cf7d313305902bd21f2ca034b5fb79ecb3c9bff2fa36888fe7a0d39432366449" exitCode=0 Jul 07 14:24:59.809627 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:59.809516 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d74p5" event={"ID":"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7","Type":"ContainerDied","Data":"cf7d313305902bd21f2ca034b5fb79ecb3c9bff2fa36888fe7a0d39432366449"} Jul 07 14:24:59.813031 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:59.812982 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" event={"ID":"48af10ba-c8eb-42d6-a322-440ed4e09d81","Type":"ContainerStarted","Data":"45b879c71e8aaea49dbdc14693dbb7ed65e88101e1fc0783b8ae1c62a04e3761"} Jul 07 14:24:59.813164 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:59.813069 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" event={"ID":"48af10ba-c8eb-42d6-a322-440ed4e09d81","Type":"ContainerStarted","Data":"d423a6c23d25eb065684cd75411db927380c58ea7d1b407e4c2bad159d1ecad0"} Jul 07 14:24:59.813164 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:59.813118 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" event={"ID":"48af10ba-c8eb-42d6-a322-440ed4e09d81","Type":"ContainerStarted","Data":"0fab91983a3696999d3ebbddce1c5a221903789f88ed9d7f5c911e3a94fbe12b"} Jul 07 14:24:59.813164 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:59.813160 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" event={"ID":"48af10ba-c8eb-42d6-a322-440ed4e09d81","Type":"ContainerStarted","Data":"87f1201eb4acb38366f2e53ad64367b99b0827449ab67eeb76f908619641ff8e"} Jul 07 14:24:59.883656 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:24:59.883619 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:00.252487 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:00.252452 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:00.334750 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:00.334704 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:00.404622 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:00.404589 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:00.632229 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:00.632182 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 07 14:25:00.700561 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:00.700526 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:00.700772 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:00.700678 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:25:00.817953 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:00.817859 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" event={"ID":"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41","Type":"ContainerStarted","Data":"b7eaa63225e37e806b76f9c14e31e70b8eaf254bd31da42d416cd445a58e382d"} Jul 07 14:25:00.819474 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:00.819444 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qch8s" event={"ID":"431a400a-c4bd-4dc1-9a6c-db75e8a4e773","Type":"ContainerStarted","Data":"12139e1a8644cb93a6b741b84de9c777376553fba14f94e017aa530e8b31f0c7"} Jul 07 14:25:00.841462 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:00.841412 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qch8s" podStartSLOduration=4.346262635 podStartE2EDuration="23.841396106s" totalImagesPullingTime="19.495133471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:00.841143137 +0000 UTC m=+24.650690629" watchObservedRunningTime="2026-07-07 14:25:00.841396106 +0000 UTC m=+24.650943596" Jul 07 14:25:00.877077 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:00.877045 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:00.967911 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:00.967865 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:01.048991 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:01.048960 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:01.228489 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:01.228405 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:01.328757 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:01.328673 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:01.429661 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:01.429628 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:01.551134 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:01.551000 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-88g99" Jul 07 14:25:01.551630 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:01.551608 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-88g99" Jul 07 14:25:01.616807 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:01.616701 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T14:25:00.632200231Z","UUID":"1c431c15-32f1-4379-9132-1db3d31ec50e","Handler":null,"Name":"","Endpoint":""} Jul 07 14:25:01.618631 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:01.618607 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 07 14:25:01.618767 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:01.618655 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 07 14:25:01.700931 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:01.700890 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:01.701104 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:01.701042 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:25:01.701104 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:01.701076 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:01.701224 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:01.701186 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:25:01.750488 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:01.750441 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:01.825777 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:01.825688 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" event={"ID":"48af10ba-c8eb-42d6-a322-440ed4e09d81","Type":"ContainerStarted","Data":"b2052410a7a45a60f032c2003b75f8ea7402360b3106222e7030f088e8bea9c3"} Jul 07 14:25:01.849871 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:01.849833 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:02.701201 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:02.701161 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:02.701410 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:02.701297 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:25:02.830855 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:02.830815 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" event={"ID":"0e3d7e6f-b33c-49a3-84dd-e9f3ec79cb41","Type":"ContainerStarted","Data":"520f42b56b2fe512e95122596b60f3dcb09aaa87b416fd8c309873f7d6ac17ae"} Jul 07 14:25:02.855159 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:02.855103 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cvgk6" podStartSLOduration=3.214146872 podStartE2EDuration="25.855084727s" totalImagesPullingTime="22.640937855s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:24:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:02.854612755 +0000 UTC m=+26.664160246" watchObservedRunningTime="2026-07-07 14:25:02.855084727 +0000 UTC m=+26.664632216" Jul 07 14:25:03.045074 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:03.045006 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8xs5n_64b3ea37-09bc-491f-8e4c-de9cc97e7b5a/dns-node-resolver/0.log" Jul 07 14:25:03.247144 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:03.247111 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:03.701215 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:03.701134 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:03.701426 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:03.701134 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:03.701426 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:03.701289 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:25:03.701426 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:03.701337 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:25:03.704034 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:03.704001 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:03.961697 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:03.961580 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:04.243582 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:04.243510 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zc4j9_b22731b9-be09-4ad5-96ae-dce04203e0ad/node-ca/0.log" Jul 07 14:25:04.701470 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:04.701450 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:04.701572 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:04.701554 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:25:04.709690 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:04.709658 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:04.842123 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:04.842092 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" event={"ID":"48af10ba-c8eb-42d6-a322-440ed4e09d81","Type":"ContainerStarted","Data":"103b344a24c9186591b4ce10ab973e73c030f295bf9dc392a8d937c7518d09a0"} Jul 07 14:25:04.842335 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:04.842317 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:25:04.842506 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:04.842485 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:25:04.857471 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:04.857445 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:25:04.857640 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:04.857626 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:25:04.880611 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:04.880561 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" podStartSLOduration=8.237010596 podStartE2EDuration="27.880546286s" totalImagesPullingTime="19.64353569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:04.87895739 +0000 UTC m=+28.688504880" watchObservedRunningTime="2026-07-07 14:25:04.880546286 +0000 UTC m=+28.690093776" Jul 07 14:25:05.309317 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:05.309059 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-88g99" Jul 07 14:25:05.310163 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:05.309343 2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 07 14:25:05.310163 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:05.309778 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-88g99" Jul 07 14:25:05.383004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:05.382969 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:05.700933 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:05.700851 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:05.701083 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:05.700851 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:05.701083 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:05.700973 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:25:05.701083 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:05.701063 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:25:05.794204 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:05.794167 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:05.845527 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:05.845494 2562 generic.go:356] "Generic (PLEG): container finished" podID="8db8a8fe-45de-4a15-93f4-bd9d3c2019c7" containerID="f9ef8e36953ea122b6b6fadef563addd8f3a3df72ab298db3ad8fd3ac494355b" exitCode=0 Jul 07 14:25:05.845706 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:05.845584 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d74p5" event={"ID":"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7","Type":"ContainerDied","Data":"f9ef8e36953ea122b6b6fadef563addd8f3a3df72ab298db3ad8fd3ac494355b"} Jul 07 14:25:05.845770 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:05.845734 2562 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 07 14:25:05.880084 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:05.880044 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:06.023716 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:06.023633 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:06.131568 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:06.131528 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:06.231570 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:06.231533 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:06.321808 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:06.321734 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:25:06.701443 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:06.701349 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:06.701582 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:06.701468 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:25:07.159854 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:07.159778 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:07.552299 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:07.552263 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:07.701271 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:07.701190 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:07.701443 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:07.701287 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:25:07.701443 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:07.701344 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:07.701521 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:07.701452 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:25:07.750677 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:07.750644 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:07.851302 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:07.851266 2562 generic.go:356] "Generic (PLEG): container finished" podID="8db8a8fe-45de-4a15-93f4-bd9d3c2019c7" containerID="52f96772ac4f87af00ad8d28e1e2ad98022c32e402f2a73745e7065f76cdd97d" exitCode=0 Jul 07 14:25:07.851472 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:07.851358 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d74p5" event={"ID":"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7","Type":"ContainerDied","Data":"52f96772ac4f87af00ad8d28e1e2ad98022c32e402f2a73745e7065f76cdd97d"} Jul 07 14:25:07.953730 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:07.953644 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:08.147713 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:08.147682 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:08.388757 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:08.388682 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:08.489938 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:08.489908 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:08.704136 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:08.701332 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:08.704136 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:08.701647 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:25:09.034663 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:09.034588 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:09.097577 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:09.097541 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:09.342589 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:09.342563 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:09.544630 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:09.544597 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:09.700701 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:09.700669 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:09.700849 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:09.700669 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:09.700849 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:09.700780 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:25:09.700849 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:09.700835 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:25:09.742412 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:09.742388 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:09.858391 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:09.858346 2562 generic.go:356] "Generic (PLEG): container finished" podID="8db8a8fe-45de-4a15-93f4-bd9d3c2019c7" containerID="9cfd9305321a53a86418d2316e00871d54ffc7eb18d78abb68ac6c5f99914232" exitCode=0 Jul 07 14:25:09.858527 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:09.858426 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d74p5" event={"ID":"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7","Type":"ContainerDied","Data":"9cfd9305321a53a86418d2316e00871d54ffc7eb18d78abb68ac6c5f99914232"} Jul 07 14:25:09.940297 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:09.940266 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:10.041575 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:10.041496 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:10.146910 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:10.146874 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:10.420016 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:10.419986 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs\") pod \"network-metrics-daemon-4bgbc\" (UID: \"74893d67-336e-4e2b-9782-8f0e68db8463\") " pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:10.420180 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:10.420023 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret\") pod \"global-pull-secret-syncer-7w875\" (UID: \"fdf0d65e-e82f-4183-90df-fde1f0ce3435\") " pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:10.420180 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:10.420055 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkk7\" (UniqueName: \"kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7\") pod \"network-check-target-6b55m\" (UID: \"337cda64-fcd6-481c-86ed-b25d2396aaca\") " pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:10.420180 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:10.420137 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:25:10.420180 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:10.420149 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 07 14:25:10.420180 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:10.420162 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 07 14:25:10.420180 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:10.420171 2562 projected.go:196] Error preparing data for projected volume kube-api-access-xvkk7 for pod openshift-network-diagnostics/network-check-target-6b55m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:25:10.420350 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:10.420193 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs podName:74893d67-336e-4e2b-9782-8f0e68db8463 nodeName:}" failed. No retries permitted until 2026-07-07 14:25:42.420178154 +0000 UTC m=+66.229725622 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs") pod "network-metrics-daemon-4bgbc" (UID: "74893d67-336e-4e2b-9782-8f0e68db8463") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 07 14:25:10.420350 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:10.420139 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 07 14:25:10.420350 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:10.420212 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7 podName:337cda64-fcd6-481c-86ed-b25d2396aaca nodeName:}" failed. No retries permitted until 2026-07-07 14:25:42.420202697 +0000 UTC m=+66.229750171 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xvkk7" (UniqueName: "kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7") pod "network-check-target-6b55m" (UID: "337cda64-fcd6-481c-86ed-b25d2396aaca") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 07 14:25:10.420350 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:10.420225 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret podName:fdf0d65e-e82f-4183-90df-fde1f0ce3435 nodeName:}" failed. No retries permitted until 2026-07-07 14:25:42.42021853 +0000 UTC m=+66.229765998 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret") pod "global-pull-secret-syncer-7w875" (UID: "fdf0d65e-e82f-4183-90df-fde1f0ce3435") : object "kube-system"/"original-pull-secret" not registered Jul 07 14:25:10.701000 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:10.700921 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:10.701130 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:10.701039 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:25:10.942704 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:10.942669 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:11.041875 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:11.041789 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:11.552339 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:11.552303 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:11.654664 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:11.654627 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:11.701189 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:11.701159 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:11.701394 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:11.701281 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:25:11.701394 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:11.701334 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:11.701518 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:11.701444 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:25:11.754364 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:11.754329 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:12.142956 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:12.142918 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:12.347083 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:12.347043 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:12.701098 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:12.701062 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:12.701314 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:12.701186 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:25:12.949261 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:12.949215 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:13.348502 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:13.348468 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:13.449827 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:13.449792 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:13.550024 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:13.549988 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:13.700910 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:13.700872 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:13.701097 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:13.700919 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:13.701097 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:13.701015 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:25:13.701185 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:13.701149 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:25:13.942987 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:13.942949 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:14.551512 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:14.551471 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:14.701173 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:14.701146 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:14.701312 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:14.701287 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:25:14.757618 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:14.757568 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:14.873224 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:14.873190 2562 generic.go:356] "Generic (PLEG): container finished" podID="8db8a8fe-45de-4a15-93f4-bd9d3c2019c7" containerID="65c0d2702575918e1423bd837335b31a910786c921b6b81724cd2857ddb88475" exitCode=0 Jul 07 14:25:14.873391 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:14.873261 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d74p5" event={"ID":"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7","Type":"ContainerDied","Data":"65c0d2702575918e1423bd837335b31a910786c921b6b81724cd2857ddb88475"} Jul 07 14:25:14.893931 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:14.893889 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:14.994014 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:14.993986 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:15.104654 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:15.104559 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:15.198874 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:15.198839 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:15.344323 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:15.344286 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:15.560009 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:15.559976 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:15.701450 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:15.701419 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:15.701450 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:15.701446 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:15.701677 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:15.701535 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:25:15.701677 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:15.701627 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:25:15.755703 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:15.755663 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:15.877891 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:15.877813 2562 generic.go:356] "Generic (PLEG): container finished" podID="8db8a8fe-45de-4a15-93f4-bd9d3c2019c7" containerID="318fa829b304d8449656d5f9c4bc3c7a311c34cfd08fcb30b0547219407be37a" exitCode=0 Jul 07 14:25:15.877891 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:15.877869 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d74p5" event={"ID":"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7","Type":"ContainerDied","Data":"318fa829b304d8449656d5f9c4bc3c7a311c34cfd08fcb30b0547219407be37a"} Jul 07 14:25:15.951200 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:15.951171 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:16.058529 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:16.058473 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:16.176401 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:16.176357 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:16.266549 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:16.266516 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:16.389853 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:16.389799 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:16.701246 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:16.701215 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:16.701604 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:16.701310 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:25:16.783390 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:16.783351 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:16.883548 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:16.883519 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d74p5" event={"ID":"8db8a8fe-45de-4a15-93f4-bd9d3c2019c7","Type":"ContainerStarted","Data":"86fd42ee59c965a92f2dde225d15bd8726345169b137b163a55de9b63a4a0f77"} Jul 07 14:25:16.909273 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:16.909222 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d74p5" podStartSLOduration=6.597232809 podStartE2EDuration="39.909206922s" totalImagesPullingTime="33.311974113s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:24:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:16.90863548 +0000 UTC m=+40.718182972" watchObservedRunningTime="2026-07-07 14:25:16.909206922 +0000 UTC m=+40.718754412" Jul 07 14:25:16.978554 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:16.978462 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:17.701198 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:17.701170 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:17.701348 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:17.701170 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:17.701348 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:17.701268 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:25:17.701348 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:17.701327 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:25:18.012169 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:18.012073 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:18.111967 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:18.111925 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:18.701065 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:18.701023 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:18.701338 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:18.701177 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:25:19.701317 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:19.701277 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:19.701806 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:19.701290 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:19.701806 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:19.701433 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:25:19.701806 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:19.701465 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:25:20.701203 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:20.701172 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:20.701393 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:20.701286 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:25:21.700729 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:21.700693 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:21.700729 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:21.700714 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:21.700954 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:21.700803 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:25:21.700954 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:21.700916 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:25:22.701184 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:22.701146 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:22.701688 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:22.701260 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:25:23.700975 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:23.700946 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:23.701164 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:23.700946 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:23.701164 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:23.701052 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:25:23.701164 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:23.701140 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:25:24.701496 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:24.701461 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:24.701869 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:24.701602 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:25:25.164170 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:25.164137 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:25.251125 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:25.251091 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:25.700643 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:25.700613 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:25.700831 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:25.700611 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:25.700831 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:25.700729 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:25:25.700831 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:25.700775 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:25:25.848251 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:25.848214 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:25.944000 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:25.943964 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:26.702092 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:26.702055 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:26.702269 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:26.702148 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:25:27.134725 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:27.134641 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:27.312657 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:27.312620 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:27.701063 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:27.701025 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:27.701276 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:27.701028 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:27.701276 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:27.701128 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:25:27.701276 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:27.701210 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:25:27.867057 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:27.867024 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:28.144618 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:28.144537 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:28.245722 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:28.245693 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:28.345187 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:28.345151 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:28.435215 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:28.435187 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:28.520271 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:28.520236 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:28.700986 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:28.700905 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:28.701131 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:28.701024 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:25:29.503825 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:29.503650 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4bgbc"] Jul 07 14:25:29.504601 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:29.503902 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:29.504601 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:29.503990 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:25:29.506752 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:29.506724 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6b55m"] Jul 07 14:25:29.506913 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:29.506860 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:29.506990 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:29.506952 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:25:29.507410 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:29.507367 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7w875"] Jul 07 14:25:29.507533 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:29.507487 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:29.507612 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:29.507584 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:25:30.054662 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:30.054633 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:30.124317 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:30.124281 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:30.701164 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:30.701136 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:30.701678 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:30.701267 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:25:30.701678 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:30.701315 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:30.701678 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:30.701450 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:25:30.846042 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:30.846008 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:30.950486 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:30.950457 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:31.701478 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:31.701448 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:31.701932 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:31.701543 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6b55m" podUID="337cda64-fcd6-481c-86ed-b25d2396aaca" Jul 07 14:25:32.613092 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:32.613060 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:32.678256 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:32.678227 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:32.701315 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:32.701274 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:32.701483 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:32.701413 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7w875" podUID="fdf0d65e-e82f-4183-90df-fde1f0ce3435" Jul 07 14:25:32.701483 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:32.701470 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:32.701867 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:32.701600 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4bgbc" podUID="74893d67-336e-4e2b-9782-8f0e68db8463" Jul 07 14:25:33.524545 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.524521 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-34.ec2.internal" event="NodeReady" Jul 07 14:25:33.524745 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.524642 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 07 14:25:33.583799 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.583766 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-btsnj"] Jul 07 14:25:33.585574 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.585558 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-btsnj" Jul 07 14:25:33.587947 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.587914 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 07 14:25:33.588188 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.588167 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 07 14:25:33.588298 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.588186 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fpbkf\"" Jul 07 14:25:33.588298 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.588224 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 07 14:25:33.589777 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.589758 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mhwfk"] Jul 07 14:25:33.591529 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.591510 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mhwfk" Jul 07 14:25:33.594853 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.594833 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 07 14:25:33.594853 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.594838 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 07 14:25:33.595059 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.594876 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 07 14:25:33.595117 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.595078 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 07 14:25:33.595552 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.595319 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5ms4k\"" Jul 07 14:25:33.597636 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.597616 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-btsnj"] Jul 07 14:25:33.603944 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.603781 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mhwfk"] Jul 07 14:25:33.613493 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.613474 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d45ft"] Jul 07 14:25:33.615269 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.615254 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d45ft" Jul 07 14:25:33.617405 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.617363 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 07 14:25:33.617405 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.617397 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 07 14:25:33.617589 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.617475 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-82rv6\"" Jul 07 14:25:33.631702 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.631685 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d45ft"] Jul 07 14:25:33.676078 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.676044 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d642794f-888e-40d7-9d1c-ded85fa8153f-crio-socket\") pod \"insights-runtime-extractor-mhwfk\" (UID: \"d642794f-888e-40d7-9d1c-ded85fa8153f\") " pod="openshift-insights/insights-runtime-extractor-mhwfk" Jul 07 14:25:33.676250 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.676091 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d642794f-888e-40d7-9d1c-ded85fa8153f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mhwfk\" (UID: \"d642794f-888e-40d7-9d1c-ded85fa8153f\") " pod="openshift-insights/insights-runtime-extractor-mhwfk" Jul 07 14:25:33.676250 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.676118 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9b2e306-1295-4e9f-8455-dbaa646ccae0-config-volume\") pod \"dns-default-d45ft\" (UID: \"e9b2e306-1295-4e9f-8455-dbaa646ccae0\") " pod="openshift-dns/dns-default-d45ft" Jul 07 14:25:33.676250 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.676180 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmkkn\" (UniqueName: \"kubernetes.io/projected/da4cacc9-a04a-49e6-aef4-7417f001222c-kube-api-access-nmkkn\") pod \"ingress-canary-btsnj\" (UID: \"da4cacc9-a04a-49e6-aef4-7417f001222c\") " pod="openshift-ingress-canary/ingress-canary-btsnj" Jul 07 14:25:33.676250 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.676208 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d642794f-888e-40d7-9d1c-ded85fa8153f-data-volume\") pod \"insights-runtime-extractor-mhwfk\" (UID: \"d642794f-888e-40d7-9d1c-ded85fa8153f\") " pod="openshift-insights/insights-runtime-extractor-mhwfk" Jul 07 14:25:33.676250 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.676234 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da4cacc9-a04a-49e6-aef4-7417f001222c-cert\") pod \"ingress-canary-btsnj\" (UID: \"da4cacc9-a04a-49e6-aef4-7417f001222c\") " pod="openshift-ingress-canary/ingress-canary-btsnj" Jul 07 14:25:33.676430 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.676275 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d642794f-888e-40d7-9d1c-ded85fa8153f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mhwfk\" (UID: \"d642794f-888e-40d7-9d1c-ded85fa8153f\") " pod="openshift-insights/insights-runtime-extractor-mhwfk" Jul 07 14:25:33.676430 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.676294 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9b2e306-1295-4e9f-8455-dbaa646ccae0-tmp-dir\") pod \"dns-default-d45ft\" (UID: \"e9b2e306-1295-4e9f-8455-dbaa646ccae0\") " pod="openshift-dns/dns-default-d45ft" Jul 07 14:25:33.676430 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.676334 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9b2e306-1295-4e9f-8455-dbaa646ccae0-metrics-tls\") pod \"dns-default-d45ft\" (UID: \"e9b2e306-1295-4e9f-8455-dbaa646ccae0\") " pod="openshift-dns/dns-default-d45ft" Jul 07 14:25:33.676430 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.676388 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kgf5\" (UniqueName: \"kubernetes.io/projected/d642794f-888e-40d7-9d1c-ded85fa8153f-kube-api-access-7kgf5\") pod \"insights-runtime-extractor-mhwfk\" (UID: \"d642794f-888e-40d7-9d1c-ded85fa8153f\") " pod="openshift-insights/insights-runtime-extractor-mhwfk" Jul 07 14:25:33.676430 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.676406 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndw6d\" (UniqueName: \"kubernetes.io/projected/e9b2e306-1295-4e9f-8455-dbaa646ccae0-kube-api-access-ndw6d\") pod \"dns-default-d45ft\" (UID: \"e9b2e306-1295-4e9f-8455-dbaa646ccae0\") " pod="openshift-dns/dns-default-d45ft" Jul 07 14:25:33.701204 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.701166 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:33.703983 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.703962 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 14:25:33.706022 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.706001 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 14:25:33.707340 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.707323 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rzdvx\"" Jul 07 14:25:33.776697 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.776613 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kgf5\" (UniqueName: \"kubernetes.io/projected/d642794f-888e-40d7-9d1c-ded85fa8153f-kube-api-access-7kgf5\") pod \"insights-runtime-extractor-mhwfk\" (UID: \"d642794f-888e-40d7-9d1c-ded85fa8153f\") " pod="openshift-insights/insights-runtime-extractor-mhwfk" Jul 07 14:25:33.776697 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.776650 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndw6d\" (UniqueName: \"kubernetes.io/projected/e9b2e306-1295-4e9f-8455-dbaa646ccae0-kube-api-access-ndw6d\") pod \"dns-default-d45ft\" (UID: \"e9b2e306-1295-4e9f-8455-dbaa646ccae0\") " pod="openshift-dns/dns-default-d45ft" Jul 07 14:25:33.776697 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.776673 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d642794f-888e-40d7-9d1c-ded85fa8153f-crio-socket\") pod \"insights-runtime-extractor-mhwfk\" (UID: \"d642794f-888e-40d7-9d1c-ded85fa8153f\") " pod="openshift-insights/insights-runtime-extractor-mhwfk" Jul 07 14:25:33.776697 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.776701 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d642794f-888e-40d7-9d1c-ded85fa8153f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mhwfk\" (UID: \"d642794f-888e-40d7-9d1c-ded85fa8153f\") " pod="openshift-insights/insights-runtime-extractor-mhwfk" Jul 07 14:25:33.776939 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.776887 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9b2e306-1295-4e9f-8455-dbaa646ccae0-config-volume\") pod \"dns-default-d45ft\" (UID: \"e9b2e306-1295-4e9f-8455-dbaa646ccae0\") " pod="openshift-dns/dns-default-d45ft" Jul 07 14:25:33.776939 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.776934 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmkkn\" (UniqueName: \"kubernetes.io/projected/da4cacc9-a04a-49e6-aef4-7417f001222c-kube-api-access-nmkkn\") pod \"ingress-canary-btsnj\" (UID: \"da4cacc9-a04a-49e6-aef4-7417f001222c\") " pod="openshift-ingress-canary/ingress-canary-btsnj" Jul 07 14:25:33.777024 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.776958 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d642794f-888e-40d7-9d1c-ded85fa8153f-data-volume\") pod \"insights-runtime-extractor-mhwfk\" (UID: \"d642794f-888e-40d7-9d1c-ded85fa8153f\") " pod="openshift-insights/insights-runtime-extractor-mhwfk" Jul 07 14:25:33.777024 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.776988 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da4cacc9-a04a-49e6-aef4-7417f001222c-cert\") pod \"ingress-canary-btsnj\" (UID: \"da4cacc9-a04a-49e6-aef4-7417f001222c\") " pod="openshift-ingress-canary/ingress-canary-btsnj" Jul 07 14:25:33.777116 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.777036 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d642794f-888e-40d7-9d1c-ded85fa8153f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mhwfk\" (UID: \"d642794f-888e-40d7-9d1c-ded85fa8153f\") " pod="openshift-insights/insights-runtime-extractor-mhwfk" Jul 07 14:25:33.777116 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.777036 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d642794f-888e-40d7-9d1c-ded85fa8153f-crio-socket\") pod \"insights-runtime-extractor-mhwfk\" (UID: \"d642794f-888e-40d7-9d1c-ded85fa8153f\") " pod="openshift-insights/insights-runtime-extractor-mhwfk" Jul 07 14:25:33.777116 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.777059 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9b2e306-1295-4e9f-8455-dbaa646ccae0-tmp-dir\") pod \"dns-default-d45ft\" (UID: \"e9b2e306-1295-4e9f-8455-dbaa646ccae0\") " pod="openshift-dns/dns-default-d45ft" Jul 07 14:25:33.777116 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.777092 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9b2e306-1295-4e9f-8455-dbaa646ccae0-metrics-tls\") pod \"dns-default-d45ft\" (UID: \"e9b2e306-1295-4e9f-8455-dbaa646ccae0\") " pod="openshift-dns/dns-default-d45ft" Jul 07 14:25:33.777363 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.777341 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d642794f-888e-40d7-9d1c-ded85fa8153f-data-volume\") pod \"insights-runtime-extractor-mhwfk\" (UID: \"d642794f-888e-40d7-9d1c-ded85fa8153f\") " pod="openshift-insights/insights-runtime-extractor-mhwfk" Jul 07 14:25:33.777541 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.777515 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9b2e306-1295-4e9f-8455-dbaa646ccae0-tmp-dir\") pod \"dns-default-d45ft\" (UID: \"e9b2e306-1295-4e9f-8455-dbaa646ccae0\") " pod="openshift-dns/dns-default-d45ft" Jul 07 14:25:33.777613 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.777582 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e9b2e306-1295-4e9f-8455-dbaa646ccae0-config-volume\") pod \"dns-default-d45ft\" (UID: \"e9b2e306-1295-4e9f-8455-dbaa646ccae0\") " pod="openshift-dns/dns-default-d45ft" Jul 07 14:25:33.777676 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.777659 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d642794f-888e-40d7-9d1c-ded85fa8153f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mhwfk\" (UID: \"d642794f-888e-40d7-9d1c-ded85fa8153f\") " pod="openshift-insights/insights-runtime-extractor-mhwfk" Jul 07 14:25:33.780771 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.780749 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d642794f-888e-40d7-9d1c-ded85fa8153f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mhwfk\" (UID: \"d642794f-888e-40d7-9d1c-ded85fa8153f\") " pod="openshift-insights/insights-runtime-extractor-mhwfk" Jul 07 14:25:33.780866 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.780754 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9b2e306-1295-4e9f-8455-dbaa646ccae0-metrics-tls\") pod \"dns-default-d45ft\" (UID: \"e9b2e306-1295-4e9f-8455-dbaa646ccae0\") " pod="openshift-dns/dns-default-d45ft" Jul 07 14:25:33.780866 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.780799 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da4cacc9-a04a-49e6-aef4-7417f001222c-cert\") pod \"ingress-canary-btsnj\" (UID: \"da4cacc9-a04a-49e6-aef4-7417f001222c\") " pod="openshift-ingress-canary/ingress-canary-btsnj" Jul 07 14:25:33.786161 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.786140 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndw6d\" (UniqueName: \"kubernetes.io/projected/e9b2e306-1295-4e9f-8455-dbaa646ccae0-kube-api-access-ndw6d\") pod \"dns-default-d45ft\" (UID: \"e9b2e306-1295-4e9f-8455-dbaa646ccae0\") " pod="openshift-dns/dns-default-d45ft" Jul 07 14:25:33.786366 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.786351 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kgf5\" (UniqueName: \"kubernetes.io/projected/d642794f-888e-40d7-9d1c-ded85fa8153f-kube-api-access-7kgf5\") pod \"insights-runtime-extractor-mhwfk\" (UID: \"d642794f-888e-40d7-9d1c-ded85fa8153f\") " pod="openshift-insights/insights-runtime-extractor-mhwfk" Jul 07 14:25:33.787278 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.787257 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmkkn\" (UniqueName: \"kubernetes.io/projected/da4cacc9-a04a-49e6-aef4-7417f001222c-kube-api-access-nmkkn\") pod \"ingress-canary-btsnj\" (UID: \"da4cacc9-a04a-49e6-aef4-7417f001222c\") " pod="openshift-ingress-canary/ingress-canary-btsnj" Jul 07 14:25:33.812074 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.812047 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:33.876931 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.876906 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:33.895885 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.895859 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-btsnj" Jul 07 14:25:33.904428 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.904407 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mhwfk" Jul 07 14:25:33.923619 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:33.923592 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d45ft" Jul 07 14:25:34.064097 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:34.064063 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-btsnj"] Jul 07 14:25:34.069504 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:25:34.069475 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4cacc9_a04a_49e6_aef4_7417f001222c.slice/crio-896c0b61d2d585f89e5b581462ded08503e7ecd678ce453726e8dae0531151eb WatchSource:0}: Error finding container 896c0b61d2d585f89e5b581462ded08503e7ecd678ce453726e8dae0531151eb: Status 404 returned error can't find the container with id 896c0b61d2d585f89e5b581462ded08503e7ecd678ce453726e8dae0531151eb Jul 07 14:25:34.084390 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:34.084356 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mhwfk"] Jul 07 14:25:34.088251 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:25:34.088227 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd642794f_888e_40d7_9d1c_ded85fa8153f.slice/crio-16416f79480443dc79bc1156908a3d5f07583fb7983a0d922760719b8548a085 WatchSource:0}: Error finding container 16416f79480443dc79bc1156908a3d5f07583fb7983a0d922760719b8548a085: Status 404 returned error can't find the container with id 16416f79480443dc79bc1156908a3d5f07583fb7983a0d922760719b8548a085 Jul 07 14:25:34.090415 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:34.090394 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d45ft"] Jul 07 14:25:34.093437 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:25:34.093399 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9b2e306_1295_4e9f_8455_dbaa646ccae0.slice/crio-322e598459c765184f36111e8be9740ca5f8a9d47918cbdf370ef81528e81698 WatchSource:0}: Error finding container 322e598459c765184f36111e8be9740ca5f8a9d47918cbdf370ef81528e81698: Status 404 returned error can't find the container with id 322e598459c765184f36111e8be9740ca5f8a9d47918cbdf370ef81528e81698 Jul 07 14:25:34.212737 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:34.212707 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:34.279718 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:34.279690 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:34.701520 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:34.701490 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:34.701689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:34.701676 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:34.704903 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:34.704559 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 14:25:34.707774 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:34.705897 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p9bwj\"" Jul 07 14:25:34.707774 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:34.706118 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 14:25:34.929719 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:34.929616 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d45ft" event={"ID":"e9b2e306-1295-4e9f-8455-dbaa646ccae0","Type":"ContainerStarted","Data":"322e598459c765184f36111e8be9740ca5f8a9d47918cbdf370ef81528e81698"} Jul 07 14:25:34.932283 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:34.932184 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mhwfk" event={"ID":"d642794f-888e-40d7-9d1c-ded85fa8153f","Type":"ContainerStarted","Data":"16416f79480443dc79bc1156908a3d5f07583fb7983a0d922760719b8548a085"} Jul 07 14:25:34.934558 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:34.934491 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-btsnj" event={"ID":"da4cacc9-a04a-49e6-aef4-7417f001222c","Type":"ContainerStarted","Data":"896c0b61d2d585f89e5b581462ded08503e7ecd678ce453726e8dae0531151eb"} Jul 07 14:25:37.060005 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:37.059967 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:37.163167 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:37.163137 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:37.863779 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:37.863742 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zzfcn" Jul 07 14:25:37.980447 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:37.980411 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:38.091112 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.091073 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:38.209339 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.209296 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:38.353361 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.353330 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:38.454243 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.454207 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:38.469457 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.469362 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7"] Jul 07 14:25:38.472579 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.472557 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7" Jul 07 14:25:38.475271 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.475021 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5q5pv\"" Jul 07 14:25:38.475271 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.475033 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 07 14:25:38.475271 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.475071 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 07 14:25:38.475271 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.475021 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 07 14:25:38.475650 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.475387 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 07 14:25:38.475725 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.475689 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 07 14:25:38.487034 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.487008 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7"] Jul 07 14:25:38.517801 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.517768 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhncm\" (UniqueName: \"kubernetes.io/projected/cc790513-59cf-4bc9-ab29-63daebcb0ed4-kube-api-access-rhncm\") pod \"prometheus-operator-567f7b8b7c-smsz7\" (UID: \"cc790513-59cf-4bc9-ab29-63daebcb0ed4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7" Jul 07 14:25:38.517964 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.517840 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc790513-59cf-4bc9-ab29-63daebcb0ed4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-smsz7\" (UID: \"cc790513-59cf-4bc9-ab29-63daebcb0ed4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7" Jul 07 14:25:38.517964 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.517868 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc790513-59cf-4bc9-ab29-63daebcb0ed4-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-smsz7\" (UID: \"cc790513-59cf-4bc9-ab29-63daebcb0ed4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7" Jul 07 14:25:38.517964 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.517916 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc790513-59cf-4bc9-ab29-63daebcb0ed4-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-smsz7\" (UID: \"cc790513-59cf-4bc9-ab29-63daebcb0ed4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7" Jul 07 14:25:38.573618 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.573581 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:38.618779 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.618731 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc790513-59cf-4bc9-ab29-63daebcb0ed4-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-smsz7\" (UID: \"cc790513-59cf-4bc9-ab29-63daebcb0ed4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7" Jul 07 14:25:38.618940 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.618811 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc790513-59cf-4bc9-ab29-63daebcb0ed4-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-smsz7\" (UID: \"cc790513-59cf-4bc9-ab29-63daebcb0ed4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7" Jul 07 14:25:38.618940 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.618855 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhncm\" (UniqueName: \"kubernetes.io/projected/cc790513-59cf-4bc9-ab29-63daebcb0ed4-kube-api-access-rhncm\") pod \"prometheus-operator-567f7b8b7c-smsz7\" (UID: \"cc790513-59cf-4bc9-ab29-63daebcb0ed4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7" Jul 07 14:25:38.619095 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.619058 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc790513-59cf-4bc9-ab29-63daebcb0ed4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-smsz7\" (UID: \"cc790513-59cf-4bc9-ab29-63daebcb0ed4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7" Jul 07 14:25:38.619481 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.619463 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc790513-59cf-4bc9-ab29-63daebcb0ed4-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-smsz7\" (UID: \"cc790513-59cf-4bc9-ab29-63daebcb0ed4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7" Jul 07 14:25:38.621574 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.621550 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cc790513-59cf-4bc9-ab29-63daebcb0ed4-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-smsz7\" (UID: \"cc790513-59cf-4bc9-ab29-63daebcb0ed4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7" Jul 07 14:25:38.621681 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.621664 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc790513-59cf-4bc9-ab29-63daebcb0ed4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-smsz7\" (UID: \"cc790513-59cf-4bc9-ab29-63daebcb0ed4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7" Jul 07 14:25:38.631309 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.631286 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhncm\" (UniqueName: \"kubernetes.io/projected/cc790513-59cf-4bc9-ab29-63daebcb0ed4-kube-api-access-rhncm\") pod \"prometheus-operator-567f7b8b7c-smsz7\" (UID: \"cc790513-59cf-4bc9-ab29-63daebcb0ed4\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7" Jul 07 14:25:38.782805 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:38.782722 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7" Jul 07 14:25:39.466999 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:39.466875 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7"] Jul 07 14:25:39.493413 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:39.493324 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:39.876268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:39.876230 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:39.950152 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:39.950081 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mhwfk" event={"ID":"d642794f-888e-40d7-9d1c-ded85fa8153f","Type":"ContainerStarted","Data":"b841c065734a7f6334b801f2c34d4af3593c70bfbb6e17670473b79e9e5f354a"} Jul 07 14:25:39.952169 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:39.952138 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-btsnj" event={"ID":"da4cacc9-a04a-49e6-aef4-7417f001222c","Type":"ContainerStarted","Data":"7f5f7c3da282856148837a27c17607cbe422e4d7e9ae0a95bbe4ce9aecbef3a1"} Jul 07 14:25:39.954112 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:39.954072 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7" event={"ID":"cc790513-59cf-4bc9-ab29-63daebcb0ed4","Type":"ContainerStarted","Data":"0807e16ee6871064808a87d2f2bf80f6291950839bca670b2715a991c457f4c2"} Jul 07 14:25:39.956399 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:39.956293 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d45ft" event={"ID":"e9b2e306-1295-4e9f-8455-dbaa646ccae0","Type":"ContainerStarted","Data":"0af012d145c6f48013e7595ed3504e9dbc19b2319f79d016dc91c7b24653583d"} Jul 07 14:25:39.956399 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:39.956348 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d45ft" event={"ID":"e9b2e306-1295-4e9f-8455-dbaa646ccae0","Type":"ContainerStarted","Data":"66987e30817c7e760a98c985f580fca38f2ba1c6b19a96d4d27904a6d9dce45e"} Jul 07 14:25:39.956640 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:39.956600 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d45ft" Jul 07 14:25:39.973157 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:39.973042 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-btsnj" podStartSLOduration=1.72772737 podStartE2EDuration="6.973020934s" totalImagesPullingTime="5.245293564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:39.971875451 +0000 UTC m=+63.781422937" watchObservedRunningTime="2026-07-07 14:25:39.973020934 +0000 UTC m=+63.782568427" Jul 07 14:25:39.974967 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:39.974925 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:39.996110 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:39.995722 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-d45ft" podStartSLOduration=1.780729908 podStartE2EDuration="6.995698741s" totalImagesPullingTime="5.214968833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:39.99447425 +0000 UTC m=+63.804021741" watchObservedRunningTime="2026-07-07 14:25:39.995698741 +0000 UTC m=+63.805246234" Jul 07 14:25:40.962697 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:40.962657 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7" event={"ID":"cc790513-59cf-4bc9-ab29-63daebcb0ed4","Type":"ContainerStarted","Data":"f7632a44a9941ea71b0a798bd1c608081e9241c3375acb05402bba6c6f40b715"} Jul 07 14:25:40.963186 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:40.962719 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7" event={"ID":"cc790513-59cf-4bc9-ab29-63daebcb0ed4","Type":"ContainerStarted","Data":"3a0945ecafee21649f1c1e050f0c0f4c3b7e83c06bd4d6339bd89e3b2dd4960d"} Jul 07 14:25:40.965023 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:40.964995 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mhwfk" event={"ID":"d642794f-888e-40d7-9d1c-ded85fa8153f","Type":"ContainerStarted","Data":"58a5c936e8e7433e993c4f304d2b539af3e2da3dbe2ef5722241f97572b79e87"} Jul 07 14:25:41.021560 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:41.021525 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:41.438108 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:41.438073 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:41.510409 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:41.510352 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:41.825757 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:41.825659 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:41.887763 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:41.887725 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:42.449478 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.449439 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs\") pod \"network-metrics-daemon-4bgbc\" (UID: \"74893d67-336e-4e2b-9782-8f0e68db8463\") " pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:42.449926 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.449493 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret\") pod \"global-pull-secret-syncer-7w875\" (UID: \"fdf0d65e-e82f-4183-90df-fde1f0ce3435\") " pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:42.449926 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.449548 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvkk7\" (UniqueName: \"kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7\") pod \"network-check-target-6b55m\" (UID: \"337cda64-fcd6-481c-86ed-b25d2396aaca\") " pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:42.452846 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.452818 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 07 14:25:42.453889 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.453865 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 07 14:25:42.453889 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.453885 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 07 14:25:42.462549 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.462523 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fdf0d65e-e82f-4183-90df-fde1f0ce3435-original-pull-secret\") pod \"global-pull-secret-syncer-7w875\" (UID: \"fdf0d65e-e82f-4183-90df-fde1f0ce3435\") " pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:42.462685 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.462569 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74893d67-336e-4e2b-9782-8f0e68db8463-metrics-certs\") pod \"network-metrics-daemon-4bgbc\" (UID: \"74893d67-336e-4e2b-9782-8f0e68db8463\") " pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:42.462830 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.462812 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 07 14:25:42.473431 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.473407 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvkk7\" (UniqueName: \"kubernetes.io/projected/337cda64-fcd6-481c-86ed-b25d2396aaca-kube-api-access-xvkk7\") pod \"network-check-target-6b55m\" (UID: \"337cda64-fcd6-481c-86ed-b25d2396aaca\") " pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:42.519408 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.519356 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7w875" Jul 07 14:25:42.533368 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.533339 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-p9bwj\"" Jul 07 14:25:42.540797 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.540766 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4bgbc" Jul 07 14:25:42.616360 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.616328 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:42.710430 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.710321 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:42.712330 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.712306 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rzdvx\"" Jul 07 14:25:42.719297 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.719276 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:42.844912 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.844878 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:42.899410 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.898593 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-smsz7" podStartSLOduration=3.753021681 podStartE2EDuration="4.89856752s" totalImagesPullingTime="1.145545839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:25:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:40.987194494 +0000 UTC m=+64.796741985" watchObservedRunningTime="2026-07-07 14:25:42.89856752 +0000 UTC m=+66.708114992" Jul 07 14:25:42.899410 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.898846 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc"] Jul 07 14:25:42.902592 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.902547 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc" Jul 07 14:25:42.906454 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.906067 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 07 14:25:42.906454 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.906297 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-xrxtv\"" Jul 07 14:25:42.907530 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.906822 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 07 14:25:42.916761 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.916686 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc"] Jul 07 14:25:42.922467 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.922127 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7wxcn"] Jul 07 14:25:42.927169 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.926298 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:42.929656 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.929500 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 07 14:25:42.929767 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.929720 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 07 14:25:42.929828 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.929766 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 07 14:25:42.929956 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.929937 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p2mqk\"" Jul 07 14:25:42.935439 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.934427 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66"] Jul 07 14:25:42.936959 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.936909 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:42.940686 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.940650 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 07 14:25:42.940836 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.940818 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 07 14:25:42.940915 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.940887 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-4p4wv\"" Jul 07 14:25:42.940915 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.940902 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 07 14:25:42.943104 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.943078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:42.951886 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.951859 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-textfile\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:42.952010 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.951905 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:42.952010 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.951940 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-wtmp\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:42.952010 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.951977 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-root\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:42.952010 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.951998 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/15739c79-527a-466c-98b1-0fa07eb88e62-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-t6gwc\" (UID: \"15739c79-527a-466c-98b1-0fa07eb88e62\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc" Jul 07 14:25:42.952207 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.952016 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c55k\" (UniqueName: \"kubernetes.io/projected/15739c79-527a-466c-98b1-0fa07eb88e62-kube-api-access-6c55k\") pod \"openshift-state-metrics-54cf96dfcf-t6gwc\" (UID: \"15739c79-527a-466c-98b1-0fa07eb88e62\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc" Jul 07 14:25:42.952207 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.952040 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15739c79-527a-466c-98b1-0fa07eb88e62-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-t6gwc\" (UID: \"15739c79-527a-466c-98b1-0fa07eb88e62\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc" Jul 07 14:25:42.952207 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.952060 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7dcq\" (UniqueName: \"kubernetes.io/projected/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-kube-api-access-p7dcq\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:42.952207 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.952092 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-accelerators-collector-config\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:42.952207 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.952128 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-tls\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:42.952207 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.952152 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-sys\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:42.952498 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.952385 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-metrics-client-ca\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:42.952630 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.952608 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15739c79-527a-466c-98b1-0fa07eb88e62-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-t6gwc\" (UID: \"15739c79-527a-466c-98b1-0fa07eb88e62\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc" Jul 07 14:25:42.954112 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.954039 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66"] Jul 07 14:25:42.976970 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:42.976894 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mhwfk" event={"ID":"d642794f-888e-40d7-9d1c-ded85fa8153f","Type":"ContainerStarted","Data":"485c93323c8e1d99e9d39120cc9ee29a8459201925d485df1c0587d73af8e27d"} Jul 07 14:25:43.003975 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.003943 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6b55m"] Jul 07 14:25:43.006941 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:25:43.006916 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337cda64_fcd6_481c_86ed_b25d2396aaca.slice/crio-702af08179e39aad1227777aaa8601c604775da279ac893b79f18bccdebe4d3a WatchSource:0}: Error finding container 702af08179e39aad1227777aaa8601c604775da279ac893b79f18bccdebe4d3a: Status 404 returned error can't find the container with id 702af08179e39aad1227777aaa8601c604775da279ac893b79f18bccdebe4d3a Jul 07 14:25:43.026291 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.026181 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4bgbc"] Jul 07 14:25:43.026987 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:25:43.026961 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74893d67_336e_4e2b_9782_8f0e68db8463.slice/crio-f8d71fcc51ed606f989adbe948a27a0284e8982c560ffb5b3f316a4f4a6bdb0c WatchSource:0}: Error finding container f8d71fcc51ed606f989adbe948a27a0284e8982c560ffb5b3f316a4f4a6bdb0c: Status 404 returned error can't find the container with id f8d71fcc51ed606f989adbe948a27a0284e8982c560ffb5b3f316a4f4a6bdb0c Jul 07 14:25:43.028261 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.028238 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:43.035484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:25:43.035461 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf0d65e_e82f_4183_90df_fde1f0ce3435.slice/crio-8bd2600f5c3d07b35fc137f6641834f563608a896e4da2ce13decc4d7f92cc1f WatchSource:0}: Error finding container 8bd2600f5c3d07b35fc137f6641834f563608a896e4da2ce13decc4d7f92cc1f: Status 404 returned error can't find the container with id 8bd2600f5c3d07b35fc137f6641834f563608a896e4da2ce13decc4d7f92cc1f Jul 07 14:25:43.036069 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.036046 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7w875"] Jul 07 14:25:43.053984 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.053956 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15739c79-527a-466c-98b1-0fa07eb88e62-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-t6gwc\" (UID: \"15739c79-527a-466c-98b1-0fa07eb88e62\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc" Jul 07 14:25:43.054091 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.053989 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7dcq\" (UniqueName: \"kubernetes.io/projected/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-kube-api-access-p7dcq\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:43.054091 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.054013 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-accelerators-collector-config\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:43.054163 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.054127 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26tzd\" (UniqueName: \"kubernetes.io/projected/734e668f-6e19-4fbc-989e-6cdcb9a25532-kube-api-access-26tzd\") pod \"kube-state-metrics-7c9dbb8b7d-s2w66\" (UID: \"734e668f-6e19-4fbc-989e-6cdcb9a25532\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.054212 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.054174 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-tls\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:43.054212 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.054204 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/734e668f-6e19-4fbc-989e-6cdcb9a25532-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-s2w66\" (UID: \"734e668f-6e19-4fbc-989e-6cdcb9a25532\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.054309 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.054236 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-sys\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:43.054309 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:43.054273 2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 07 14:25:43.054422 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.054319 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-sys\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:43.054422 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:43.054338 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-tls podName:48fa86c9-ae09-45d8-bbdc-558a49fcb7dc nodeName:}" failed. No retries permitted until 2026-07-07 14:25:43.554317113 +0000 UTC m=+67.363864595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-tls") pod "node-exporter-7wxcn" (UID: "48fa86c9-ae09-45d8-bbdc-558a49fcb7dc") : secret "node-exporter-tls" not found Jul 07 14:25:43.054422 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.054268 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-metrics-client-ca\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:43.054422 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.054405 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15739c79-527a-466c-98b1-0fa07eb88e62-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-t6gwc\" (UID: \"15739c79-527a-466c-98b1-0fa07eb88e62\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc" Jul 07 14:25:43.054622 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.054550 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-textfile\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:43.054622 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.054589 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/734e668f-6e19-4fbc-989e-6cdcb9a25532-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-s2w66\" (UID: \"734e668f-6e19-4fbc-989e-6cdcb9a25532\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.054710 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.054621 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:43.054710 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.054653 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-wtmp\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:43.054818 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.054799 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-wtmp\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:43.054875 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.054857 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/734e668f-6e19-4fbc-989e-6cdcb9a25532-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-s2w66\" (UID: \"734e668f-6e19-4fbc-989e-6cdcb9a25532\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.054927 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.054882 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-metrics-client-ca\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:43.054927 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.054891 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/734e668f-6e19-4fbc-989e-6cdcb9a25532-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-s2w66\" (UID: \"734e668f-6e19-4fbc-989e-6cdcb9a25532\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.054927 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.054895 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-accelerators-collector-config\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:43.054927 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.054907 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-textfile\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:43.055126 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.055046 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-root\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:43.055126 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.055086 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/15739c79-527a-466c-98b1-0fa07eb88e62-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-t6gwc\" (UID: \"15739c79-527a-466c-98b1-0fa07eb88e62\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc" Jul 07 14:25:43.055126 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.055114 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c55k\" (UniqueName: \"kubernetes.io/projected/15739c79-527a-466c-98b1-0fa07eb88e62-kube-api-access-6c55k\") pod \"openshift-state-metrics-54cf96dfcf-t6gwc\" (UID: \"15739c79-527a-466c-98b1-0fa07eb88e62\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc" Jul 07 14:25:43.055279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.055141 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/734e668f-6e19-4fbc-989e-6cdcb9a25532-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-s2w66\" (UID: \"734e668f-6e19-4fbc-989e-6cdcb9a25532\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.055279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.055089 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-root\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:43.055279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.055182 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/15739c79-527a-466c-98b1-0fa07eb88e62-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-t6gwc\" (UID: \"15739c79-527a-466c-98b1-0fa07eb88e62\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc" Jul 07 14:25:43.056953 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.056935 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/15739c79-527a-466c-98b1-0fa07eb88e62-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-t6gwc\" (UID: \"15739c79-527a-466c-98b1-0fa07eb88e62\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc" Jul 07 14:25:43.057032 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.056993 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:43.057261 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.057244 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/15739c79-527a-466c-98b1-0fa07eb88e62-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-t6gwc\" (UID: \"15739c79-527a-466c-98b1-0fa07eb88e62\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc" Jul 07 14:25:43.070860 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.070837 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c55k\" (UniqueName: \"kubernetes.io/projected/15739c79-527a-466c-98b1-0fa07eb88e62-kube-api-access-6c55k\") pod \"openshift-state-metrics-54cf96dfcf-t6gwc\" (UID: \"15739c79-527a-466c-98b1-0fa07eb88e62\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc" Jul 07 14:25:43.070860 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.070855 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7dcq\" (UniqueName: \"kubernetes.io/projected/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-kube-api-access-p7dcq\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:43.084865 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.084821 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mhwfk" podStartSLOduration=1.471783487 podStartE2EDuration="10.084805462s" totalImagesPullingTime="8.613021975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:25:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:43.084540397 +0000 UTC m=+66.894087888" watchObservedRunningTime="2026-07-07 14:25:43.084805462 +0000 UTC m=+66.894352956" Jul 07 14:25:43.155572 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.155540 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26tzd\" (UniqueName: \"kubernetes.io/projected/734e668f-6e19-4fbc-989e-6cdcb9a25532-kube-api-access-26tzd\") pod \"kube-state-metrics-7c9dbb8b7d-s2w66\" (UID: \"734e668f-6e19-4fbc-989e-6cdcb9a25532\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.155766 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.155591 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/734e668f-6e19-4fbc-989e-6cdcb9a25532-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-s2w66\" (UID: \"734e668f-6e19-4fbc-989e-6cdcb9a25532\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.155766 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.155626 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/734e668f-6e19-4fbc-989e-6cdcb9a25532-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-s2w66\" (UID: \"734e668f-6e19-4fbc-989e-6cdcb9a25532\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.155766 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.155654 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/734e668f-6e19-4fbc-989e-6cdcb9a25532-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-s2w66\" (UID: \"734e668f-6e19-4fbc-989e-6cdcb9a25532\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.155766 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.155672 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/734e668f-6e19-4fbc-989e-6cdcb9a25532-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-s2w66\" (UID: \"734e668f-6e19-4fbc-989e-6cdcb9a25532\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.155766 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.155696 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/734e668f-6e19-4fbc-989e-6cdcb9a25532-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-s2w66\" (UID: \"734e668f-6e19-4fbc-989e-6cdcb9a25532\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.156009 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:43.155784 2562 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 07 14:25:43.156009 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:43.155854 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/734e668f-6e19-4fbc-989e-6cdcb9a25532-kube-state-metrics-tls podName:734e668f-6e19-4fbc-989e-6cdcb9a25532 nodeName:}" failed. No retries permitted until 2026-07-07 14:25:43.655838362 +0000 UTC m=+67.465385831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/734e668f-6e19-4fbc-989e-6cdcb9a25532-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-s2w66" (UID: "734e668f-6e19-4fbc-989e-6cdcb9a25532") : secret "kube-state-metrics-tls" not found Jul 07 14:25:43.156129 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.156101 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/734e668f-6e19-4fbc-989e-6cdcb9a25532-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-s2w66\" (UID: \"734e668f-6e19-4fbc-989e-6cdcb9a25532\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.156424 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.156395 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/734e668f-6e19-4fbc-989e-6cdcb9a25532-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-s2w66\" (UID: \"734e668f-6e19-4fbc-989e-6cdcb9a25532\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.156507 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.156488 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/734e668f-6e19-4fbc-989e-6cdcb9a25532-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-s2w66\" (UID: \"734e668f-6e19-4fbc-989e-6cdcb9a25532\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.158262 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.158241 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/734e668f-6e19-4fbc-989e-6cdcb9a25532-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-s2w66\" (UID: \"734e668f-6e19-4fbc-989e-6cdcb9a25532\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.165680 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.165660 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26tzd\" (UniqueName: \"kubernetes.io/projected/734e668f-6e19-4fbc-989e-6cdcb9a25532-kube-api-access-26tzd\") pod \"kube-state-metrics-7c9dbb8b7d-s2w66\" (UID: \"734e668f-6e19-4fbc-989e-6cdcb9a25532\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.227224 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.227145 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc" Jul 07 14:25:43.252722 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.252688 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:43.355291 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.355266 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc"] Jul 07 14:25:43.357596 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:25:43.357567 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15739c79_527a_466c_98b1_0fa07eb88e62.slice/crio-96a366706c994c0b8df14b87b62c3b233305f03d2b051ff6b71d0e2e6a393bd3 WatchSource:0}: Error finding container 96a366706c994c0b8df14b87b62c3b233305f03d2b051ff6b71d0e2e6a393bd3: Status 404 returned error can't find the container with id 96a366706c994c0b8df14b87b62c3b233305f03d2b051ff6b71d0e2e6a393bd3 Jul 07 14:25:43.468695 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.468662 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:43.559821 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.559724 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-tls\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:43.560092 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:43.559878 2562 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 07 14:25:43.560092 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:43.559947 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-tls podName:48fa86c9-ae09-45d8-bbdc-558a49fcb7dc nodeName:}" failed. No retries permitted until 2026-07-07 14:25:44.559928291 +0000 UTC m=+68.369475762 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-tls") pod "node-exporter-7wxcn" (UID: "48fa86c9-ae09-45d8-bbdc-558a49fcb7dc") : secret "node-exporter-tls" not found Jul 07 14:25:43.572544 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.572491 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:43.661231 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.661085 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/734e668f-6e19-4fbc-989e-6cdcb9a25532-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-s2w66\" (UID: \"734e668f-6e19-4fbc-989e-6cdcb9a25532\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.664018 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.663988 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/734e668f-6e19-4fbc-989e-6cdcb9a25532-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-s2w66\" (UID: \"734e668f-6e19-4fbc-989e-6cdcb9a25532\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.673774 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.673680 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:43.770661 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.770610 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:43.847218 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.846886 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" Jul 07 14:25:43.899164 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.899123 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:43.991345 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.991266 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6b55m" event={"ID":"337cda64-fcd6-481c-86ed-b25d2396aaca","Type":"ContainerStarted","Data":"702af08179e39aad1227777aaa8601c604775da279ac893b79f18bccdebe4d3a"} Jul 07 14:25:43.994276 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.994242 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7w875" event={"ID":"fdf0d65e-e82f-4183-90df-fde1f0ce3435","Type":"ContainerStarted","Data":"8bd2600f5c3d07b35fc137f6641834f563608a896e4da2ce13decc4d7f92cc1f"} Jul 07 14:25:43.996322 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:43.996243 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4bgbc" event={"ID":"74893d67-336e-4e2b-9782-8f0e68db8463","Type":"ContainerStarted","Data":"f8d71fcc51ed606f989adbe948a27a0284e8982c560ffb5b3f316a4f4a6bdb0c"} Jul 07 14:25:44.001007 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.000971 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc" event={"ID":"15739c79-527a-466c-98b1-0fa07eb88e62","Type":"ContainerStarted","Data":"196d21c542c36d96adbb7608d386e551e301e78f38b853357e38f6421e464756"} Jul 07 14:25:44.001140 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.001049 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc" event={"ID":"15739c79-527a-466c-98b1-0fa07eb88e62","Type":"ContainerStarted","Data":"9cd3a3619a1b7e4f9b3e0abb798d8fa3e4d5d156780f42c84b6d046bd9f6595d"} Jul 07 14:25:44.001140 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.001077 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc" event={"ID":"15739c79-527a-466c-98b1-0fa07eb88e62","Type":"ContainerStarted","Data":"96a366706c994c0b8df14b87b62c3b233305f03d2b051ff6b71d0e2e6a393bd3"} Jul 07 14:25:44.043309 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.043274 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66"] Jul 07 14:25:44.048033 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.048005 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:44.066698 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.066165 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 14:25:44.071867 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.071769 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.075474 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.074736 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 07 14:25:44.075474 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.074962 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 07 14:25:44.075474 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.075290 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 07 14:25:44.075474 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.075309 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 07 14:25:44.075474 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.075432 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 07 14:25:44.076764 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.075546 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 07 14:25:44.076764 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.075634 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 07 14:25:44.076764 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.075687 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6gxmb\"" Jul 07 14:25:44.076764 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.075733 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 07 14:25:44.076764 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.075812 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 07 14:25:44.089038 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.088959 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 14:25:44.166181 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.166145 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.166402 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.166195 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.166402 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.166228 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pdvm\" (UniqueName: \"kubernetes.io/projected/c2dff414-68a7-4ae9-8599-3e50e17acfbe-kube-api-access-7pdvm\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.166402 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.166259 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2dff414-68a7-4ae9-8599-3e50e17acfbe-config-out\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.166402 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.166287 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.166402 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.166330 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.166402 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.166385 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2dff414-68a7-4ae9-8599-3e50e17acfbe-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.166734 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.166413 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-config-volume\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.166734 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.166438 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c2dff414-68a7-4ae9-8599-3e50e17acfbe-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.166734 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.166518 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2dff414-68a7-4ae9-8599-3e50e17acfbe-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.166734 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.166545 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.166734 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.166585 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-web-config\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.166734 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.166610 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2dff414-68a7-4ae9-8599-3e50e17acfbe-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.222894 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.222807 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:44.268622 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.267688 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2dff414-68a7-4ae9-8599-3e50e17acfbe-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.268622 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.267731 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-config-volume\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.268622 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.267759 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c2dff414-68a7-4ae9-8599-3e50e17acfbe-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.268622 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.267829 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2dff414-68a7-4ae9-8599-3e50e17acfbe-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.268622 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.267856 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.268622 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.267881 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-web-config\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.268622 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.267909 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2dff414-68a7-4ae9-8599-3e50e17acfbe-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.268622 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.267939 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.268622 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.267965 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.268622 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.267999 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pdvm\" (UniqueName: \"kubernetes.io/projected/c2dff414-68a7-4ae9-8599-3e50e17acfbe-kube-api-access-7pdvm\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.268622 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.268031 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2dff414-68a7-4ae9-8599-3e50e17acfbe-config-out\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.268622 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.268061 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.268622 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.268108 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.269403 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.269021 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c2dff414-68a7-4ae9-8599-3e50e17acfbe-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.269403 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:25:44.269264 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c2dff414-68a7-4ae9-8599-3e50e17acfbe-alertmanager-trusted-ca-bundle podName:c2dff414-68a7-4ae9-8599-3e50e17acfbe nodeName:}" failed. No retries permitted until 2026-07-07 14:25:44.769242028 +0000 UTC m=+68.578789500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/c2dff414-68a7-4ae9-8599-3e50e17acfbe-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "c2dff414-68a7-4ae9-8599-3e50e17acfbe") : configmap references non-existent config key: ca-bundle.crt Jul 07 14:25:44.270894 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.270797 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c2dff414-68a7-4ae9-8599-3e50e17acfbe-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.275077 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.274642 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.275077 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.274823 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2dff414-68a7-4ae9-8599-3e50e17acfbe-config-out\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.275077 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.274989 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-config-volume\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.275077 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.275029 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2dff414-68a7-4ae9-8599-3e50e17acfbe-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.275658 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.275419 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-web-config\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.275658 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.275524 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.281829 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.277829 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.281829 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.278730 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.286054 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.285988 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c2dff414-68a7-4ae9-8599-3e50e17acfbe-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.289274 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.289226 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pdvm\" (UniqueName: \"kubernetes.io/projected/c2dff414-68a7-4ae9-8599-3e50e17acfbe-kube-api-access-7pdvm\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.570893 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.570792 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-tls\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:44.573901 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.573836 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48fa86c9-ae09-45d8-bbdc-558a49fcb7dc-node-exporter-tls\") pod \"node-exporter-7wxcn\" (UID: \"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc\") " pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:44.738779 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.738553 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7wxcn" Jul 07 14:25:44.751903 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:25:44.751871 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fa86c9_ae09_45d8_bbdc_558a49fcb7dc.slice/crio-1f8fc6631a6867e605b30b97eb4a599c32c59a919f83265aea7c018c38ad667f WatchSource:0}: Error finding container 1f8fc6631a6867e605b30b97eb4a599c32c59a919f83265aea7c018c38ad667f: Status 404 returned error can't find the container with id 1f8fc6631a6867e605b30b97eb4a599c32c59a919f83265aea7c018c38ad667f Jul 07 14:25:44.773410 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.772584 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2dff414-68a7-4ae9-8599-3e50e17acfbe-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.773742 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.773683 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c2dff414-68a7-4ae9-8599-3e50e17acfbe-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c2dff414-68a7-4ae9-8599-3e50e17acfbe\") " pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:44.985303 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:44.985237 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 07 14:25:45.007416 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.007302 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wxcn" event={"ID":"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc","Type":"ContainerStarted","Data":"1f8fc6631a6867e605b30b97eb4a599c32c59a919f83265aea7c018c38ad667f"} Jul 07 14:25:45.011412 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.011270 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4bgbc" event={"ID":"74893d67-336e-4e2b-9782-8f0e68db8463","Type":"ContainerStarted","Data":"43d8617cc3021b79dfe618e0c721175fa26850f479c5bc5ae348f2f23cf0baa3"} Jul 07 14:25:45.011412 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.011339 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4bgbc" event={"ID":"74893d67-336e-4e2b-9782-8f0e68db8463","Type":"ContainerStarted","Data":"dad9724df1b71fdfa3ad5004171d9d69fa593a814063fcf8b3d73b696e395188"} Jul 07 14:25:45.013900 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.013805 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" event={"ID":"734e668f-6e19-4fbc-989e-6cdcb9a25532","Type":"ContainerStarted","Data":"972f0739e3f50744b8130ffb3b12d46d2d0967e141a150205b83b2fef56d3823"} Jul 07 14:25:45.019102 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.018002 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-848b6bcc56-l8q8m"] Jul 07 14:25:45.023484 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.023449 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc" event={"ID":"15739c79-527a-466c-98b1-0fa07eb88e62","Type":"ContainerStarted","Data":"c6a25711d6e67c7b05e84a7081a3e3606b2cf4bbe9e627d184d45080ec6c7f77"} Jul 07 14:25:45.023822 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.023802 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.027510 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.026925 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-atm8rrtj2if24\"" Jul 07 14:25:45.027510 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.027196 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 07 14:25:45.027510 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.027221 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 07 14:25:45.027510 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.027491 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 07 14:25:45.028152 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.027881 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-phdw6\"" Jul 07 14:25:45.028152 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.027937 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 07 14:25:45.028152 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.027881 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 07 14:25:45.039515 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.039121 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4bgbc" podStartSLOduration=66.363521218 podStartE2EDuration="1m8.039098076s" totalImagesPullingTime="1.675576858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:45.036348395 +0000 UTC m=+68.845895896" watchObservedRunningTime="2026-07-07 14:25:45.039098076 +0000 UTC m=+68.848645566" Jul 07 14:25:45.041293 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.041203 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-848b6bcc56-l8q8m"] Jul 07 14:25:45.066402 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.066090 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-t6gwc" podStartSLOduration=1.815674836 podStartE2EDuration="3.066067952s" totalImagesPullingTime="1.250393116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:45.065577519 +0000 UTC m=+68.875125010" watchObservedRunningTime="2026-07-07 14:25:45.066067952 +0000 UTC m=+68.875615443" Jul 07 14:25:45.078012 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.077131 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ace66132-5583-448f-9dea-9d1704bb2b18-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.078012 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.077174 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ace66132-5583-448f-9dea-9d1704bb2b18-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.078012 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.077208 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ace66132-5583-448f-9dea-9d1704bb2b18-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.078012 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.077269 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ace66132-5583-448f-9dea-9d1704bb2b18-metrics-client-ca\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.078012 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.077330 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ace66132-5583-448f-9dea-9d1704bb2b18-secret-grpc-tls\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.078012 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.077357 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ace66132-5583-448f-9dea-9d1704bb2b18-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.078012 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.077408 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ace66132-5583-448f-9dea-9d1704bb2b18-secret-thanos-querier-tls\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.078012 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.077452 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdx6l\" (UniqueName: \"kubernetes.io/projected/ace66132-5583-448f-9dea-9d1704bb2b18-kube-api-access-gdx6l\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.178661 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.177826 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ace66132-5583-448f-9dea-9d1704bb2b18-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.178661 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.177882 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ace66132-5583-448f-9dea-9d1704bb2b18-metrics-client-ca\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.178661 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.177921 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ace66132-5583-448f-9dea-9d1704bb2b18-secret-grpc-tls\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.178661 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.177948 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ace66132-5583-448f-9dea-9d1704bb2b18-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.178661 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.178002 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ace66132-5583-448f-9dea-9d1704bb2b18-secret-thanos-querier-tls\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.178661 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.178038 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdx6l\" (UniqueName: \"kubernetes.io/projected/ace66132-5583-448f-9dea-9d1704bb2b18-kube-api-access-gdx6l\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.178661 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.178094 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ace66132-5583-448f-9dea-9d1704bb2b18-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.178661 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.178127 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ace66132-5583-448f-9dea-9d1704bb2b18-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.180964 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.180808 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ace66132-5583-448f-9dea-9d1704bb2b18-metrics-client-ca\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.181824 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.181661 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 07 14:25:45.184112 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.184018 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ace66132-5583-448f-9dea-9d1704bb2b18-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.184112 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.184090 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ace66132-5583-448f-9dea-9d1704bb2b18-secret-thanos-querier-tls\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.185586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.185515 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ace66132-5583-448f-9dea-9d1704bb2b18-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.185979 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.185741 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ace66132-5583-448f-9dea-9d1704bb2b18-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.186553 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.186485 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ace66132-5583-448f-9dea-9d1704bb2b18-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.187219 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.187196 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ace66132-5583-448f-9dea-9d1704bb2b18-secret-grpc-tls\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.191710 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.191683 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:45.192695 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.192648 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdx6l\" (UniqueName: \"kubernetes.io/projected/ace66132-5583-448f-9dea-9d1704bb2b18-kube-api-access-gdx6l\") pod \"thanos-querier-848b6bcc56-l8q8m\" (UID: \"ace66132-5583-448f-9dea-9d1704bb2b18\") " pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.265762 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.265725 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:45.339139 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.338791 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:45.544058 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.543934 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-848b6bcc56-l8q8m"] Jul 07 14:25:45.861717 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:25:45.861647 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace66132_5583_448f_9dea_9d1704bb2b18.slice/crio-09d3e0681d61d495b23c0593a841f9837e8f03ec960dedd320edbc2bd490315c WatchSource:0}: Error finding container 09d3e0681d61d495b23c0593a841f9837e8f03ec960dedd320edbc2bd490315c: Status 404 returned error can't find the container with id 09d3e0681d61d495b23c0593a841f9837e8f03ec960dedd320edbc2bd490315c Jul 07 14:25:45.981627 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:45.981540 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:46.025835 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:46.025740 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" event={"ID":"ace66132-5583-448f-9dea-9d1704bb2b18","Type":"ContainerStarted","Data":"09d3e0681d61d495b23c0593a841f9837e8f03ec960dedd320edbc2bd490315c"} Jul 07 14:25:46.027218 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:46.027183 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2dff414-68a7-4ae9-8599-3e50e17acfbe","Type":"ContainerStarted","Data":"c263f0ff0de10bcc6f1eaab529a209477248bd9cb47121f5e7b14421f7a6bbeb"} Jul 07 14:25:46.071914 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:46.071826 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:46.194949 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:46.194862 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:46.299131 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:46.299079 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:47.648099 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:47.648057 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:47.746579 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:47.746533 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:47.834018 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:47.833978 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:49.041743 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.041705 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" event={"ID":"734e668f-6e19-4fbc-989e-6cdcb9a25532","Type":"ContainerStarted","Data":"03dddb693710b737087c9679425970a64232bd5cb37b203b93955856ef0066cc"} Jul 07 14:25:49.043515 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.043480 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6b55m" event={"ID":"337cda64-fcd6-481c-86ed-b25d2396aaca","Type":"ContainerStarted","Data":"38a5652a12026079b9a4da10faa96de9622ef2b061d6510645e6c0fe8c105e22"} Jul 07 14:25:49.043660 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.043582 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:25:49.045572 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.045541 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wxcn" event={"ID":"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc","Type":"ContainerStarted","Data":"9d33260985100f3aeb2d543b465dc693660886c0f2bac90cadbc9e378c96a52d"} Jul 07 14:25:49.062838 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.062535 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6b55m" podStartSLOduration=66.308568404 podStartE2EDuration="1m12.062516915s" totalImagesPullingTime="5.753948511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:49.060835645 +0000 UTC m=+72.870383137" watchObservedRunningTime="2026-07-07 14:25:49.062516915 +0000 UTC m=+72.872064405" Jul 07 14:25:49.330940 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.330806 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:49.332501 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.332472 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:25:49.362389 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.362345 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.364237 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.364201 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:25:49.364935 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.364907 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 07 14:25:49.365164 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.365145 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 07 14:25:49.365787 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.365751 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 07 14:25:49.368794 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.367701 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 07 14:25:49.368794 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.367771 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 07 14:25:49.368794 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.367705 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 07 14:25:49.368794 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.367773 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 07 14:25:49.368794 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.367969 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 07 14:25:49.368794 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.367969 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 07 14:25:49.368794 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.368039 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 07 14:25:49.368794 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.368089 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e3mjkeni2baiu\"" Jul 07 14:25:49.368794 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.368257 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 07 14:25:49.368794 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.368436 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 07 14:25:49.368794 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.368628 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 07 14:25:49.368794 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.368632 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pddq5\"" Jul 07 14:25:49.370340 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.370028 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 07 14:25:49.373597 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.373573 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 07 14:25:49.403596 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.403564 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:49.420179 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420132 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.420179 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420189 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.420449 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420213 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l72g4\" (UniqueName: \"kubernetes.io/projected/89ca039b-62fb-4c67-ab1c-57c07a6838c7-kube-api-access-l72g4\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.420449 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420242 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.420449 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420267 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.420449 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420301 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.420449 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420330 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-web-config\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.420449 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420358 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89ca039b-62fb-4c67-ab1c-57c07a6838c7-config-out\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.420449 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420399 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-config\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.420750 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420466 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.420750 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420571 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.420750 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420609 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.420750 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420635 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.420750 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420658 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/89ca039b-62fb-4c67-ab1c-57c07a6838c7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.420750 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420717 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.420750 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420746 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.421006 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420773 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.421006 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420800 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89ca039b-62fb-4c67-ab1c-57c07a6838c7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.421006 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420840 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.421006 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.420868 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.522395 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.522337 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89ca039b-62fb-4c67-ab1c-57c07a6838c7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.522395 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.522400 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.522665 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.522531 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.522665 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.522615 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.522665 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.522646 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.522818 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.522770 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l72g4\" (UniqueName: \"kubernetes.io/projected/89ca039b-62fb-4c67-ab1c-57c07a6838c7-kube-api-access-l72g4\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.522818 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.522809 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.522922 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.522839 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.522922 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.522882 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.522922 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.522916 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-web-config\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.523077 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.522948 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89ca039b-62fb-4c67-ab1c-57c07a6838c7-config-out\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.523077 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.522983 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-config\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.523077 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.523021 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.523077 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.523059 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.523313 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.523090 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.523313 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.523113 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.523313 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.523136 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/89ca039b-62fb-4c67-ab1c-57c07a6838c7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.523313 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.523188 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.523313 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.523223 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.523313 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.523255 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.523313 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.523268 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.523702 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.523583 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.524480 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.524116 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.524480 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.524335 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.524480 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.524356 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.525568 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.524867 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.525568 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.524967 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/89ca039b-62fb-4c67-ab1c-57c07a6838c7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.525568 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.525548 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/89ca039b-62fb-4c67-ab1c-57c07a6838c7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.527245 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.525811 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.527245 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.526333 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.528092 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.528022 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.528803 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.528270 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/89ca039b-62fb-4c67-ab1c-57c07a6838c7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.528885 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.528806 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.529148 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.528980 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.529148 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.529101 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/89ca039b-62fb-4c67-ab1c-57c07a6838c7-config-out\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.529455 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.529424 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-web-config\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.529593 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.529569 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.530737 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.530691 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.532134 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.531701 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/89ca039b-62fb-4c67-ab1c-57c07a6838c7-config\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.534511 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.534488 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l72g4\" (UniqueName: \"kubernetes.io/projected/89ca039b-62fb-4c67-ab1c-57c07a6838c7-kube-api-access-l72g4\") pod \"prometheus-k8s-0\" (UID: \"89ca039b-62fb-4c67-ab1c-57c07a6838c7\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.677957 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.677916 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:25:49.968093 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:49.968008 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d45ft" Jul 07 14:25:50.054086 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:50.054043 2562 generic.go:356] "Generic (PLEG): container finished" podID="48fa86c9-ae09-45d8-bbdc-558a49fcb7dc" containerID="9d33260985100f3aeb2d543b465dc693660886c0f2bac90cadbc9e378c96a52d" exitCode=0 Jul 07 14:25:50.055393 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:50.055142 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wxcn" event={"ID":"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc","Type":"ContainerDied","Data":"9d33260985100f3aeb2d543b465dc693660886c0f2bac90cadbc9e378c96a52d"} Jul 07 14:25:50.255805 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:50.255681 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 07 14:25:50.266477 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:25:50.261306 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ca039b_62fb_4c67_ab1c_57c07a6838c7.slice/crio-c3539d5c8ef25c20219d3ff449c35f0114c562a3b2bdb8bf915474dea4564ec1 WatchSource:0}: Error finding container c3539d5c8ef25c20219d3ff449c35f0114c562a3b2bdb8bf915474dea4564ec1: Status 404 returned error can't find the container with id c3539d5c8ef25c20219d3ff449c35f0114c562a3b2bdb8bf915474dea4564ec1 Jul 07 14:25:50.936202 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:50.936156 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:51.037843 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:51.037807 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:51.062610 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:51.062569 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" event={"ID":"734e668f-6e19-4fbc-989e-6cdcb9a25532","Type":"ContainerStarted","Data":"72990d94816b4d8fb7bb38753fdf08cb9c7c21659b1a42a099645430dc7c5aa3"} Jul 07 14:25:51.063066 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:51.062635 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" event={"ID":"734e668f-6e19-4fbc-989e-6cdcb9a25532","Type":"ContainerStarted","Data":"7864e8c85645e371d0175478f290592b5fdda8041d8fc422e19d0ffe7f5a4101"} Jul 07 14:25:51.064314 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:51.064283 2562 generic.go:356] "Generic (PLEG): container finished" podID="89ca039b-62fb-4c67-ab1c-57c07a6838c7" containerID="41850947829ee3aa5f81b746e111c8b857bb087c69ad5343a8446af189c0ac42" exitCode=0 Jul 07 14:25:51.064471 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:51.064392 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89ca039b-62fb-4c67-ab1c-57c07a6838c7","Type":"ContainerDied","Data":"41850947829ee3aa5f81b746e111c8b857bb087c69ad5343a8446af189c0ac42"} Jul 07 14:25:51.064533 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:51.064482 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89ca039b-62fb-4c67-ab1c-57c07a6838c7","Type":"ContainerStarted","Data":"c3539d5c8ef25c20219d3ff449c35f0114c562a3b2bdb8bf915474dea4564ec1"} Jul 07 14:25:51.066815 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:51.066790 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" event={"ID":"ace66132-5583-448f-9dea-9d1704bb2b18","Type":"ContainerStarted","Data":"90861fd42e7f4ee32114773ef75faef59803b1f11ce1c8f1887ba838d9283ca8"} Jul 07 14:25:51.066907 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:51.066868 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" event={"ID":"ace66132-5583-448f-9dea-9d1704bb2b18","Type":"ContainerStarted","Data":"f747f3bdf17fa64a1cb311b4b60654a31793e5e92bdf367871d7059fd017f999"} Jul 07 14:25:51.066996 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:51.066910 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" event={"ID":"ace66132-5583-448f-9dea-9d1704bb2b18","Type":"ContainerStarted","Data":"c0e64e032920062b51a4519adb7fe521fe895be3a46fc10f9baa125da9fd8625"} Jul 07 14:25:51.068584 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:51.068556 2562 generic.go:356] "Generic (PLEG): container finished" podID="c2dff414-68a7-4ae9-8599-3e50e17acfbe" containerID="8a47bc81cbbad6464bffd5bc269ea375544dc78cb7c9ec14abb5fa7bb3d1a329" exitCode=0 Jul 07 14:25:51.068667 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:51.068648 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2dff414-68a7-4ae9-8599-3e50e17acfbe","Type":"ContainerDied","Data":"8a47bc81cbbad6464bffd5bc269ea375544dc78cb7c9ec14abb5fa7bb3d1a329"} Jul 07 14:25:51.070442 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:51.070418 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7w875" event={"ID":"fdf0d65e-e82f-4183-90df-fde1f0ce3435","Type":"ContainerStarted","Data":"83da89985cc0a927833ed11dfc8326f1c82cf8c558a178e7ddebcd8d8dd329a8"} Jul 07 14:25:51.073000 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:51.072977 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wxcn" event={"ID":"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc","Type":"ContainerStarted","Data":"280552eebdb8f8f1ea28d15f29f769727d274ce381e2d98e8728cb385241a059"} Jul 07 14:25:51.073111 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:51.073020 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wxcn" event={"ID":"48fa86c9-ae09-45d8-bbdc-558a49fcb7dc","Type":"ContainerStarted","Data":"3af60a94f906677a01614137a2730b90974bf34269b3b44f7f1a42bd7f00ab58"} Jul 07 14:25:51.087111 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:51.087068 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-s2w66" podStartSLOduration=4.386300175 podStartE2EDuration="9.087053819s" totalImagesPullingTime="4.700753644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:51.086579722 +0000 UTC m=+74.896127214" watchObservedRunningTime="2026-07-07 14:25:51.087053819 +0000 UTC m=+74.896601351" Jul 07 14:25:51.110147 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:51.110091 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7wxcn" podStartSLOduration=4.991882009 podStartE2EDuration="9.110073878s" totalImagesPullingTime="3.998524875s" totalInitContainerRuntime="119.666994ms" isStatefulPod=true podCreationTimestamp="2026-07-07 14:25:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:51.108338787 +0000 UTC m=+74.917886278" watchObservedRunningTime="2026-07-07 14:25:51.110073878 +0000 UTC m=+74.919621382" Jul 07 14:25:51.226167 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:51.226109 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7w875" podStartSLOduration=67.228902151 podStartE2EDuration="1m14.22609261s" totalImagesPullingTime="6.997190459s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 14:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:51.224651302 +0000 UTC m=+75.034198786" watchObservedRunningTime="2026-07-07 14:25:51.22609261 +0000 UTC m=+75.035640099" Jul 07 14:25:52.079769 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:52.079727 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" event={"ID":"ace66132-5583-448f-9dea-9d1704bb2b18","Type":"ContainerStarted","Data":"8450e2dcf8162d04ce9de8ee15411ec74c1a04ccf68827fdad4174d5a282badc"} Jul 07 14:25:52.080186 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:52.079832 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" event={"ID":"ace66132-5583-448f-9dea-9d1704bb2b18","Type":"ContainerStarted","Data":"2846c7960bbf12b9d0c23c00a0bc413d2b04c7d43fd5e0b1d42097908bacd322"} Jul 07 14:25:52.080186 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:52.079873 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" event={"ID":"ace66132-5583-448f-9dea-9d1704bb2b18","Type":"ContainerStarted","Data":"b517086914407847b24ca456579d3f0f888b7ce1ce11ff6a0f4e28063c68788f"} Jul 07 14:25:52.108600 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:52.108180 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" podStartSLOduration=3.149891748 podStartE2EDuration="8.108156547s" totalImagesPullingTime="4.958264799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:25:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:52.107621953 +0000 UTC m=+75.917169479" watchObservedRunningTime="2026-07-07 14:25:52.108156547 +0000 UTC m=+75.917704038" Jul 07 14:25:53.087369 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:53.087330 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2dff414-68a7-4ae9-8599-3e50e17acfbe","Type":"ContainerStarted","Data":"465c5aa7840371bcb31edcfc73be1200ce1dea152005c617a7f293a64b853855"} Jul 07 14:25:53.087808 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:53.087448 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2dff414-68a7-4ae9-8599-3e50e17acfbe","Type":"ContainerStarted","Data":"4d757177698ad129f588e5fa5806a0490c2504c6d3a9718ba179853eb2a88cdd"} Jul 07 14:25:53.087808 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:53.087788 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:54.095845 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:54.095797 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2dff414-68a7-4ae9-8599-3e50e17acfbe","Type":"ContainerStarted","Data":"cae900a4d4d39b8b90c4c502164fbf90a43d7c269da850816d962acde598514f"} Jul 07 14:25:54.096320 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:54.095926 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2dff414-68a7-4ae9-8599-3e50e17acfbe","Type":"ContainerStarted","Data":"2b91a9c68d94b4f3e03992ab55ea86b9987d15306b6a1f94356c05c96f660ed7"} Jul 07 14:25:54.096320 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:54.095971 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2dff414-68a7-4ae9-8599-3e50e17acfbe","Type":"ContainerStarted","Data":"1db446695dc9b458e60d04eff1ac7c52755281ffc26df17049f94d1a4a89bcf5"} Jul 07 14:25:54.096320 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:54.096011 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c2dff414-68a7-4ae9-8599-3e50e17acfbe","Type":"ContainerStarted","Data":"3cc3fc1ed9fde5fef225d0ff90eba66c1e27874e5207937e1d004a7aa664085d"} Jul 07 14:25:54.127790 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:54.127737 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.4471012610000002 podStartE2EDuration="10.127720666s" totalImagesPullingTime="6.649940191s" totalInitContainerRuntime="30.679214ms" isStatefulPod=false podCreationTimestamp="2026-07-07 14:25:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:54.127539869 +0000 UTC m=+77.937087360" watchObservedRunningTime="2026-07-07 14:25:54.127720666 +0000 UTC m=+77.937268153" Jul 07 14:25:55.103226 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:55.103191 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89ca039b-62fb-4c67-ab1c-57c07a6838c7","Type":"ContainerStarted","Data":"6a774b30758aa01cb5be99e2750878a02d21781bb65d29c7204eb171c05ff443"} Jul 07 14:25:55.103595 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:55.103266 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89ca039b-62fb-4c67-ab1c-57c07a6838c7","Type":"ContainerStarted","Data":"a9c96b3ca6a84b984eb277b7551ac397de391cee1c963c1fedecf0da291b4b22"} Jul 07 14:25:55.103595 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:55.103295 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89ca039b-62fb-4c67-ab1c-57c07a6838c7","Type":"ContainerStarted","Data":"63541a70163d7be163c4186947e44d7c1d1d8f37bd69ee7373f32fd87e94d908"} Jul 07 14:25:55.103595 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:55.103318 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89ca039b-62fb-4c67-ab1c-57c07a6838c7","Type":"ContainerStarted","Data":"1142d559dd7b5b6a7f50358a00e91ce42ffca935446a861885a086c1616d167e"} Jul 07 14:25:55.103595 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:55.103340 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89ca039b-62fb-4c67-ab1c-57c07a6838c7","Type":"ContainerStarted","Data":"ff72af5f2080ec1b22aaced3cf589efca7bba6ffc34e30f088fc9ccabcd69e35"} Jul 07 14:25:55.103595 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:55.103363 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"89ca039b-62fb-4c67-ab1c-57c07a6838c7","Type":"ContainerStarted","Data":"4ebfcc8021b683ed8d059fefb82ea6d94f2b2cb149345d200189d372ef349693"} Jul 07 14:25:55.140252 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:55.140129 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.56187861 podStartE2EDuration="6.140105934s" totalImagesPullingTime="3.550198861s" totalInitContainerRuntime="28.028463ms" isStatefulPod=false podCreationTimestamp="2026-07-07 14:25:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:25:55.13791551 +0000 UTC m=+78.947463000" watchObservedRunningTime="2026-07-07 14:25:55.140105934 +0000 UTC m=+78.949653426" Jul 07 14:25:55.147252 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:55.147219 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:55.227260 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:55.227224 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:56.173154 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:56.173114 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:56.274010 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:56.273972 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:58.005971 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:58.005940 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:58.102473 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:58.102345 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:59.101883 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:59.101857 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-848b6bcc56-l8q8m" Jul 07 14:25:59.140110 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:59.140075 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:59.238990 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:59.238939 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:59.341128 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:59.341088 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:25:59.678651 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:25:59.678615 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:26:03.216820 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:26:03.216783 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:26:03.319661 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:26:03.319624 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:26:03.420885 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:26:03.420855 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:26:04.836157 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:26:04.836121 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:26:07.007558 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:26:07.007513 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:26:10.141459 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:26:10.141425 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:26:10.240406 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:26:10.240354 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:26:17.177266 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:26:17.177223 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:26:17.357007 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:26:17.356970 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:26:20.057958 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:26:20.057926 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6b55m" Jul 07 14:26:49.678889 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:26:49.678843 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:26:49.697721 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:26:49.697692 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:26:50.344014 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:26:50.343982 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 07 14:27:02.364507 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:27:02.364466 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:27:03.166797 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:27:03.166760 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:27:12.963196 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:27:12.963163 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:27:16.933766 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:27:16.933725 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:27:17.611997 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:27:17.611964 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:27:17.685752 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:27:17.685716 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:27:23.489068 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:27:23.489031 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:27:23.579300 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:27:23.579261 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:27:23.991466 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:27:23.991435 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:27:24.088556 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:27:24.088519 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:27:26.685827 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:27:26.685796 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:27:29.515561 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:27:29.515478 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:27:29.594523 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:27:29.594483 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:27:33.114068 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:27:33.114034 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:27:33.218210 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:27:33.218180 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:27:34.491587 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:27:34.491548 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:27:34.594812 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:27:34.594782 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:27:34.694716 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:27:34.694686 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:28:02.281936 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:28:02.281891 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:28:02.401627 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:28:02.401587 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:28:04.940577 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:28:04.940543 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:28:36.708699 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:28:36.708669 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:28:36.804450 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:28:36.804417 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:03.524329 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:03.524248 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:03.624643 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:03.624610 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:07.132064 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:07.132024 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:07.231352 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:07.231321 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:08.021446 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:08.021411 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:08.130218 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:08.130181 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:09.119589 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:09.119553 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:09.208807 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:09.208770 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:11.168042 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:11.168002 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:11.562564 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:11.562473 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:13.724390 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:13.724352 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:13.794072 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:13.794039 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:19.840901 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:19.840864 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-k7tk7"] Jul 07 14:29:19.843939 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:19.843918 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-k7tk7" Jul 07 14:29:19.847174 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:19.847130 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hkwpx\"" Jul 07 14:29:19.847939 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:19.847908 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 07 14:29:19.847939 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:19.847935 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 07 14:29:19.856124 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:19.856102 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-k7tk7"] Jul 07 14:29:19.858059 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:19.858023 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:20.007066 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:20.007027 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7db9e14e-7fe0-4a8c-a904-b3880025c665-bound-sa-token\") pod \"cert-manager-55b4d954d5-k7tk7\" (UID: \"7db9e14e-7fe0-4a8c-a904-b3880025c665\") " pod="cert-manager/cert-manager-55b4d954d5-k7tk7" Jul 07 14:29:20.007066 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:20.007063 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhd6w\" (UniqueName: \"kubernetes.io/projected/7db9e14e-7fe0-4a8c-a904-b3880025c665-kube-api-access-dhd6w\") pod \"cert-manager-55b4d954d5-k7tk7\" (UID: \"7db9e14e-7fe0-4a8c-a904-b3880025c665\") " pod="cert-manager/cert-manager-55b4d954d5-k7tk7" Jul 07 14:29:20.107965 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:20.107880 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7db9e14e-7fe0-4a8c-a904-b3880025c665-bound-sa-token\") pod \"cert-manager-55b4d954d5-k7tk7\" (UID: \"7db9e14e-7fe0-4a8c-a904-b3880025c665\") " pod="cert-manager/cert-manager-55b4d954d5-k7tk7" Jul 07 14:29:20.107965 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:20.107917 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhd6w\" (UniqueName: \"kubernetes.io/projected/7db9e14e-7fe0-4a8c-a904-b3880025c665-kube-api-access-dhd6w\") pod \"cert-manager-55b4d954d5-k7tk7\" (UID: \"7db9e14e-7fe0-4a8c-a904-b3880025c665\") " pod="cert-manager/cert-manager-55b4d954d5-k7tk7" Jul 07 14:29:20.118104 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:20.118076 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7db9e14e-7fe0-4a8c-a904-b3880025c665-bound-sa-token\") pod \"cert-manager-55b4d954d5-k7tk7\" (UID: \"7db9e14e-7fe0-4a8c-a904-b3880025c665\") " pod="cert-manager/cert-manager-55b4d954d5-k7tk7" Jul 07 14:29:20.118293 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:20.118270 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhd6w\" (UniqueName: \"kubernetes.io/projected/7db9e14e-7fe0-4a8c-a904-b3880025c665-kube-api-access-dhd6w\") pod \"cert-manager-55b4d954d5-k7tk7\" (UID: \"7db9e14e-7fe0-4a8c-a904-b3880025c665\") " pod="cert-manager/cert-manager-55b4d954d5-k7tk7" Jul 07 14:29:20.167155 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:20.167121 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-k7tk7" Jul 07 14:29:20.307617 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:20.307579 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-k7tk7"] Jul 07 14:29:20.312038 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:29:20.312003 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7db9e14e_7fe0_4a8c_a904_b3880025c665.slice/crio-e2821b13a3c30fb0ed6356ad776c1767ac78bd8f8161584a8d28b0f5180a1e9f WatchSource:0}: Error finding container e2821b13a3c30fb0ed6356ad776c1767ac78bd8f8161584a8d28b0f5180a1e9f: Status 404 returned error can't find the container with id e2821b13a3c30fb0ed6356ad776c1767ac78bd8f8161584a8d28b0f5180a1e9f Jul 07 14:29:20.912799 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:20.912764 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-k7tk7" event={"ID":"7db9e14e-7fe0-4a8c-a904-b3880025c665","Type":"ContainerStarted","Data":"e2821b13a3c30fb0ed6356ad776c1767ac78bd8f8161584a8d28b0f5180a1e9f"} Jul 07 14:29:21.014692 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:21.014639 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:21.112691 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:21.112654 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:22.398914 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:22.398878 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lxfgc"] Jul 07 14:29:22.403774 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:22.403743 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lxfgc" Jul 07 14:29:22.406616 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:22.406592 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 07 14:29:22.406761 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:22.406652 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-q59fq\"" Jul 07 14:29:22.407361 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:22.407342 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 07 14:29:22.416523 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:22.416497 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lxfgc"] Jul 07 14:29:22.528620 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:22.528580 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w47zj\" (UniqueName: \"kubernetes.io/projected/33989fec-98b5-4ae9-b339-9e835c24d617-kube-api-access-w47zj\") pod \"openshift-lws-operator-fd8ccff4c-lxfgc\" (UID: \"33989fec-98b5-4ae9-b339-9e835c24d617\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lxfgc" Jul 07 14:29:22.528813 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:22.528708 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33989fec-98b5-4ae9-b339-9e835c24d617-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lxfgc\" (UID: \"33989fec-98b5-4ae9-b339-9e835c24d617\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lxfgc" Jul 07 14:29:22.629490 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:22.629446 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w47zj\" (UniqueName: \"kubernetes.io/projected/33989fec-98b5-4ae9-b339-9e835c24d617-kube-api-access-w47zj\") pod \"openshift-lws-operator-fd8ccff4c-lxfgc\" (UID: \"33989fec-98b5-4ae9-b339-9e835c24d617\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lxfgc" Jul 07 14:29:22.629685 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:22.629512 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33989fec-98b5-4ae9-b339-9e835c24d617-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lxfgc\" (UID: \"33989fec-98b5-4ae9-b339-9e835c24d617\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lxfgc" Jul 07 14:29:22.629991 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:22.629968 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/33989fec-98b5-4ae9-b339-9e835c24d617-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lxfgc\" (UID: \"33989fec-98b5-4ae9-b339-9e835c24d617\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lxfgc" Jul 07 14:29:22.640102 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:22.640066 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w47zj\" (UniqueName: \"kubernetes.io/projected/33989fec-98b5-4ae9-b339-9e835c24d617-kube-api-access-w47zj\") pod \"openshift-lws-operator-fd8ccff4c-lxfgc\" (UID: \"33989fec-98b5-4ae9-b339-9e835c24d617\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lxfgc" Jul 07 14:29:22.715104 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:22.715022 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lxfgc" Jul 07 14:29:23.461801 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:23.461765 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lxfgc"] Jul 07 14:29:23.464368 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:29:23.464336 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33989fec_98b5_4ae9_b339_9e835c24d617.slice/crio-3eb56ce12aaca3a44689b864c2dfbf46bab6413e135fc41e3dfba4998f3fcebc WatchSource:0}: Error finding container 3eb56ce12aaca3a44689b864c2dfbf46bab6413e135fc41e3dfba4998f3fcebc: Status 404 returned error can't find the container with id 3eb56ce12aaca3a44689b864c2dfbf46bab6413e135fc41e3dfba4998f3fcebc Jul 07 14:29:23.934442 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:23.934411 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lxfgc" event={"ID":"33989fec-98b5-4ae9-b339-9e835c24d617","Type":"ContainerStarted","Data":"3eb56ce12aaca3a44689b864c2dfbf46bab6413e135fc41e3dfba4998f3fcebc"} Jul 07 14:29:23.935917 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:23.935885 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-k7tk7" event={"ID":"7db9e14e-7fe0-4a8c-a904-b3880025c665","Type":"ContainerStarted","Data":"6ddca5b86d6624f095a2dfd5b1632d58d4ca6524dcaced8b69b4c4698234faee"} Jul 07 14:29:23.954983 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:23.954924 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-k7tk7" podStartSLOduration=1.9138233869999999 podStartE2EDuration="4.95490846s" totalImagesPullingTime="3.041085073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:29:23.953663929 +0000 UTC m=+287.763211419" watchObservedRunningTime="2026-07-07 14:29:23.95490846 +0000 UTC m=+287.764455950" Jul 07 14:29:26.949920 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:26.949881 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lxfgc" event={"ID":"33989fec-98b5-4ae9-b339-9e835c24d617","Type":"ContainerStarted","Data":"82aed64c92b31cc3162c95c9ec768c148a6628ae07ab7678a6710eb10258f519"} Jul 07 14:29:26.970334 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:26.970277 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lxfgc" podStartSLOduration=2.339284236 podStartE2EDuration="4.970259262s" totalImagesPullingTime="2.630975026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:29:26.968812337 +0000 UTC m=+290.778359828" watchObservedRunningTime="2026-07-07 14:29:26.970259262 +0000 UTC m=+290.779806752" Jul 07 14:29:27.638402 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:27.638337 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:27.739646 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:27.739608 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:29.829627 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:29.829589 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:36.573358 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:36.573334 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 07 14:29:38.686080 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:38.686049 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:40.584108 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:40.584072 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:43.763309 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:43.763267 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:43.852825 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:43.852791 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:44.361537 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.361505 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:44.594277 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.594244 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:44.684315 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.684278 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:44.742984 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.742949 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb"] Jul 07 14:29:44.749796 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.749773 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" Jul 07 14:29:44.752502 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.752481 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-v7xjz\"" Jul 07 14:29:44.752749 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.752731 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 07 14:29:44.752934 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.752921 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 07 14:29:44.753008 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.752948 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 07 14:29:44.753096 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.753073 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 07 14:29:44.761937 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.761911 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb"] Jul 07 14:29:44.828243 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.828208 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64f0cd1e-7248-4696-8782-1b544a20848b-manifests\") pod \"opendatahub-operator-controller-manager-6f546bc4-m4psb\" (UID: \"64f0cd1e-7248-4696-8782-1b544a20848b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" Jul 07 14:29:44.828715 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.828263 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64f0cd1e-7248-4696-8782-1b544a20848b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f546bc4-m4psb\" (UID: \"64f0cd1e-7248-4696-8782-1b544a20848b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" Jul 07 14:29:44.828715 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.828322 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64f0cd1e-7248-4696-8782-1b544a20848b-tmp\") pod \"opendatahub-operator-controller-manager-6f546bc4-m4psb\" (UID: \"64f0cd1e-7248-4696-8782-1b544a20848b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" Jul 07 14:29:44.828715 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.828404 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-996fn\" (UniqueName: \"kubernetes.io/projected/64f0cd1e-7248-4696-8782-1b544a20848b-kube-api-access-996fn\") pod \"opendatahub-operator-controller-manager-6f546bc4-m4psb\" (UID: \"64f0cd1e-7248-4696-8782-1b544a20848b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" Jul 07 14:29:44.828715 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.828451 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64f0cd1e-7248-4696-8782-1b544a20848b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f546bc4-m4psb\" (UID: \"64f0cd1e-7248-4696-8782-1b544a20848b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" Jul 07 14:29:44.867236 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.867209 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:44.929272 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.929244 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64f0cd1e-7248-4696-8782-1b544a20848b-manifests\") pod \"opendatahub-operator-controller-manager-6f546bc4-m4psb\" (UID: \"64f0cd1e-7248-4696-8782-1b544a20848b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" Jul 07 14:29:44.929429 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.929287 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64f0cd1e-7248-4696-8782-1b544a20848b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f546bc4-m4psb\" (UID: \"64f0cd1e-7248-4696-8782-1b544a20848b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" Jul 07 14:29:44.929429 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.929320 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64f0cd1e-7248-4696-8782-1b544a20848b-tmp\") pod \"opendatahub-operator-controller-manager-6f546bc4-m4psb\" (UID: \"64f0cd1e-7248-4696-8782-1b544a20848b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" Jul 07 14:29:44.929429 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.929352 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-996fn\" (UniqueName: \"kubernetes.io/projected/64f0cd1e-7248-4696-8782-1b544a20848b-kube-api-access-996fn\") pod \"opendatahub-operator-controller-manager-6f546bc4-m4psb\" (UID: \"64f0cd1e-7248-4696-8782-1b544a20848b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" Jul 07 14:29:44.929429 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.929398 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64f0cd1e-7248-4696-8782-1b544a20848b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f546bc4-m4psb\" (UID: \"64f0cd1e-7248-4696-8782-1b544a20848b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" Jul 07 14:29:44.929736 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.929713 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/64f0cd1e-7248-4696-8782-1b544a20848b-manifests\") pod \"opendatahub-operator-controller-manager-6f546bc4-m4psb\" (UID: \"64f0cd1e-7248-4696-8782-1b544a20848b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" Jul 07 14:29:44.931776 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.931735 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64f0cd1e-7248-4696-8782-1b544a20848b-tmp\") pod \"opendatahub-operator-controller-manager-6f546bc4-m4psb\" (UID: \"64f0cd1e-7248-4696-8782-1b544a20848b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" Jul 07 14:29:44.931917 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.931806 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/64f0cd1e-7248-4696-8782-1b544a20848b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6f546bc4-m4psb\" (UID: \"64f0cd1e-7248-4696-8782-1b544a20848b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" Jul 07 14:29:44.931917 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.931856 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/64f0cd1e-7248-4696-8782-1b544a20848b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6f546bc4-m4psb\" (UID: \"64f0cd1e-7248-4696-8782-1b544a20848b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" Jul 07 14:29:44.939876 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.939814 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-996fn\" (UniqueName: \"kubernetes.io/projected/64f0cd1e-7248-4696-8782-1b544a20848b-kube-api-access-996fn\") pod \"opendatahub-operator-controller-manager-6f546bc4-m4psb\" (UID: \"64f0cd1e-7248-4696-8782-1b544a20848b\") " pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" Jul 07 14:29:44.962130 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:44.962100 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:45.062541 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:45.062506 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" Jul 07 14:29:45.106126 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:45.106096 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:45.202895 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:45.202866 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb"] Jul 07 14:29:45.203490 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:45.203468 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:45.205435 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:29:45.205416 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f0cd1e_7248_4696_8782_1b544a20848b.slice/crio-65e5b62cff95b9a1d22c2b4440b6d695d9e065e0d46a0aae4e7f73b5c79c2f52 WatchSource:0}: Error finding container 65e5b62cff95b9a1d22c2b4440b6d695d9e065e0d46a0aae4e7f73b5c79c2f52: Status 404 returned error can't find the container with id 65e5b62cff95b9a1d22c2b4440b6d695d9e065e0d46a0aae4e7f73b5c79c2f52 Jul 07 14:29:45.207197 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:45.207181 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:29:45.279361 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:45.279328 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:46.027671 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:46.027631 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" event={"ID":"64f0cd1e-7248-4696-8782-1b544a20848b","Type":"ContainerStarted","Data":"65e5b62cff95b9a1d22c2b4440b6d695d9e065e0d46a0aae4e7f73b5c79c2f52"} Jul 07 14:29:49.043922 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:49.043888 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" event={"ID":"64f0cd1e-7248-4696-8782-1b544a20848b","Type":"ContainerStarted","Data":"3540ab569c49efb2da8eb15fb02c564a67e7e097b1cdfdd202059b5f77d44d23"} Jul 07 14:29:50.050012 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:50.049978 2562 generic.go:356] "Generic (PLEG): container finished" podID="64f0cd1e-7248-4696-8782-1b544a20848b" containerID="3540ab569c49efb2da8eb15fb02c564a67e7e097b1cdfdd202059b5f77d44d23" exitCode=0 Jul 07 14:29:50.050446 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:50.050066 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" event={"ID":"64f0cd1e-7248-4696-8782-1b544a20848b","Type":"ContainerDied","Data":"3540ab569c49efb2da8eb15fb02c564a67e7e097b1cdfdd202059b5f77d44d23"} Jul 07 14:29:51.055865 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:51.055827 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" event={"ID":"64f0cd1e-7248-4696-8782-1b544a20848b","Type":"ContainerStarted","Data":"34b666c73e79eaa8aeaa831d42491c1f8833e2825829a84d4ecef69410af68b4"} Jul 07 14:29:51.056351 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:51.055928 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" Jul 07 14:29:51.080365 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:51.080313 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" podStartSLOduration=2.349176821 podStartE2EDuration="7.080296633s" totalImagesPullingTime="3.673821502s" totalInitContainerRuntime="1.05729831s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:29:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:29:51.078872757 +0000 UTC m=+314.888420248" watchObservedRunningTime="2026-07-07 14:29:51.080296633 +0000 UTC m=+314.889844122" Jul 07 14:29:56.328630 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:56.328589 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:29:56.421113 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:29:56.421077 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:01.179324 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:01.179292 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:01.322949 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:01.322913 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:02.067420 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:02.067389 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6f546bc4-m4psb" Jul 07 14:30:03.049840 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.049808 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:03.149455 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.149419 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:03.200101 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.200071 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt"] Jul 07 14:30:03.203332 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.203309 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt" Jul 07 14:30:03.205760 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.205743 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 07 14:30:03.206800 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.206784 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-vz5x5\"" Jul 07 14:30:03.206914 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.206827 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 07 14:30:03.206914 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.206872 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 07 14:30:03.214024 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.213998 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt"] Jul 07 14:30:03.291355 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.291323 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b5ab5b50-afc3-4698-8b88-7a548a0ff08c-manager-config\") pod \"lws-controller-manager-5b9884d45d-lxtjt\" (UID: \"b5ab5b50-afc3-4698-8b88-7a548a0ff08c\") " pod="openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt" Jul 07 14:30:03.291355 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.291356 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5ab5b50-afc3-4698-8b88-7a548a0ff08c-metrics-cert\") pod \"lws-controller-manager-5b9884d45d-lxtjt\" (UID: \"b5ab5b50-afc3-4698-8b88-7a548a0ff08c\") " pod="openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt" Jul 07 14:30:03.291609 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.291409 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27nbs\" (UniqueName: \"kubernetes.io/projected/b5ab5b50-afc3-4698-8b88-7a548a0ff08c-kube-api-access-27nbs\") pod \"lws-controller-manager-5b9884d45d-lxtjt\" (UID: \"b5ab5b50-afc3-4698-8b88-7a548a0ff08c\") " pod="openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt" Jul 07 14:30:03.291609 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.291442 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5ab5b50-afc3-4698-8b88-7a548a0ff08c-cert\") pod \"lws-controller-manager-5b9884d45d-lxtjt\" (UID: \"b5ab5b50-afc3-4698-8b88-7a548a0ff08c\") " pod="openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt" Jul 07 14:30:03.392694 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.392580 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b5ab5b50-afc3-4698-8b88-7a548a0ff08c-manager-config\") pod \"lws-controller-manager-5b9884d45d-lxtjt\" (UID: \"b5ab5b50-afc3-4698-8b88-7a548a0ff08c\") " pod="openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt" Jul 07 14:30:03.392694 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.392632 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5ab5b50-afc3-4698-8b88-7a548a0ff08c-metrics-cert\") pod \"lws-controller-manager-5b9884d45d-lxtjt\" (UID: \"b5ab5b50-afc3-4698-8b88-7a548a0ff08c\") " pod="openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt" Jul 07 14:30:03.392694 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.392651 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27nbs\" (UniqueName: \"kubernetes.io/projected/b5ab5b50-afc3-4698-8b88-7a548a0ff08c-kube-api-access-27nbs\") pod \"lws-controller-manager-5b9884d45d-lxtjt\" (UID: \"b5ab5b50-afc3-4698-8b88-7a548a0ff08c\") " pod="openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt" Jul 07 14:30:03.392694 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.392681 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5ab5b50-afc3-4698-8b88-7a548a0ff08c-cert\") pod \"lws-controller-manager-5b9884d45d-lxtjt\" (UID: \"b5ab5b50-afc3-4698-8b88-7a548a0ff08c\") " pod="openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt" Jul 07 14:30:03.393327 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.393304 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b5ab5b50-afc3-4698-8b88-7a548a0ff08c-manager-config\") pod \"lws-controller-manager-5b9884d45d-lxtjt\" (UID: \"b5ab5b50-afc3-4698-8b88-7a548a0ff08c\") " pod="openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt" Jul 07 14:30:03.395184 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.395158 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5ab5b50-afc3-4698-8b88-7a548a0ff08c-metrics-cert\") pod \"lws-controller-manager-5b9884d45d-lxtjt\" (UID: \"b5ab5b50-afc3-4698-8b88-7a548a0ff08c\") " pod="openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt" Jul 07 14:30:03.395184 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.395181 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5ab5b50-afc3-4698-8b88-7a548a0ff08c-cert\") pod \"lws-controller-manager-5b9884d45d-lxtjt\" (UID: \"b5ab5b50-afc3-4698-8b88-7a548a0ff08c\") " pod="openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt" Jul 07 14:30:03.404165 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.404138 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27nbs\" (UniqueName: \"kubernetes.io/projected/b5ab5b50-afc3-4698-8b88-7a548a0ff08c-kube-api-access-27nbs\") pod \"lws-controller-manager-5b9884d45d-lxtjt\" (UID: \"b5ab5b50-afc3-4698-8b88-7a548a0ff08c\") " pod="openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt" Jul 07 14:30:03.512233 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.512192 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt" Jul 07 14:30:03.658402 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:03.658364 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt"] Jul 07 14:30:03.660352 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:30:03.660328 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ab5b50_afc3_4698_8b88_7a548a0ff08c.slice/crio-b2a7867b55084c8f87ea7ca945ffae92c6a00dfdc8bbda93bb0d93178fc8938b WatchSource:0}: Error finding container b2a7867b55084c8f87ea7ca945ffae92c6a00dfdc8bbda93bb0d93178fc8938b: Status 404 returned error can't find the container with id b2a7867b55084c8f87ea7ca945ffae92c6a00dfdc8bbda93bb0d93178fc8938b Jul 07 14:30:04.116186 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:04.116103 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt" event={"ID":"b5ab5b50-afc3-4698-8b88-7a548a0ff08c","Type":"ContainerStarted","Data":"b2a7867b55084c8f87ea7ca945ffae92c6a00dfdc8bbda93bb0d93178fc8938b"} Jul 07 14:30:04.672118 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:04.672076 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:04.773218 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:04.773180 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:05.435014 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:05.434965 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:05.535957 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:05.535918 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:05.632067 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:05.632022 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:05.739014 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:05.738929 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:06.024092 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:06.024003 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:06.139902 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:06.139869 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:08.135631 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:08.135592 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt" event={"ID":"b5ab5b50-afc3-4698-8b88-7a548a0ff08c","Type":"ContainerStarted","Data":"0343c24a0c905815ad1448217baedf9aa45ff1bb96d8033b5dda27a26dd9a214"} Jul 07 14:30:08.136015 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:08.135680 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt" Jul 07 14:30:08.164750 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:08.164696 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt" podStartSLOduration=1.448264402 podStartE2EDuration="5.164681026s" totalImagesPullingTime="3.716416624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:30:08.16457807 +0000 UTC m=+331.974125571" watchObservedRunningTime="2026-07-07 14:30:08.164681026 +0000 UTC m=+331.974228529" Jul 07 14:30:10.504728 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.504695 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:10.554789 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.554748 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5f67c6995-xtj92"] Jul 07 14:30:10.558110 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.558090 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f67c6995-xtj92" Jul 07 14:30:10.561093 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.561064 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sf9lk\"" Jul 07 14:30:10.561332 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.561317 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 07 14:30:10.562080 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.562058 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 07 14:30:10.562165 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.562101 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 07 14:30:10.562165 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.562126 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 07 14:30:10.567095 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.567050 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f67c6995-xtj92"] Jul 07 14:30:10.654038 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.654001 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwghs\" (UniqueName: \"kubernetes.io/projected/7a2df886-cb1c-43fa-87ef-067cc88830b7-kube-api-access-qwghs\") pod \"kube-auth-proxy-5f67c6995-xtj92\" (UID: \"7a2df886-cb1c-43fa-87ef-067cc88830b7\") " pod="openshift-ingress/kube-auth-proxy-5f67c6995-xtj92" Jul 07 14:30:10.654235 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.654103 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2df886-cb1c-43fa-87ef-067cc88830b7-tls-certs\") pod \"kube-auth-proxy-5f67c6995-xtj92\" (UID: \"7a2df886-cb1c-43fa-87ef-067cc88830b7\") " pod="openshift-ingress/kube-auth-proxy-5f67c6995-xtj92" Jul 07 14:30:10.654235 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.654132 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a2df886-cb1c-43fa-87ef-067cc88830b7-tmp\") pod \"kube-auth-proxy-5f67c6995-xtj92\" (UID: \"7a2df886-cb1c-43fa-87ef-067cc88830b7\") " pod="openshift-ingress/kube-auth-proxy-5f67c6995-xtj92" Jul 07 14:30:10.700809 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.700739 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:10.755023 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.754938 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwghs\" (UniqueName: \"kubernetes.io/projected/7a2df886-cb1c-43fa-87ef-067cc88830b7-kube-api-access-qwghs\") pod \"kube-auth-proxy-5f67c6995-xtj92\" (UID: \"7a2df886-cb1c-43fa-87ef-067cc88830b7\") " pod="openshift-ingress/kube-auth-proxy-5f67c6995-xtj92" Jul 07 14:30:10.755169 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.755095 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2df886-cb1c-43fa-87ef-067cc88830b7-tls-certs\") pod \"kube-auth-proxy-5f67c6995-xtj92\" (UID: \"7a2df886-cb1c-43fa-87ef-067cc88830b7\") " pod="openshift-ingress/kube-auth-proxy-5f67c6995-xtj92" Jul 07 14:30:10.755169 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.755122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a2df886-cb1c-43fa-87ef-067cc88830b7-tmp\") pod \"kube-auth-proxy-5f67c6995-xtj92\" (UID: \"7a2df886-cb1c-43fa-87ef-067cc88830b7\") " pod="openshift-ingress/kube-auth-proxy-5f67c6995-xtj92" Jul 07 14:30:10.757466 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.757443 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a2df886-cb1c-43fa-87ef-067cc88830b7-tmp\") pod \"kube-auth-proxy-5f67c6995-xtj92\" (UID: \"7a2df886-cb1c-43fa-87ef-067cc88830b7\") " pod="openshift-ingress/kube-auth-proxy-5f67c6995-xtj92" Jul 07 14:30:10.757673 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.757653 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7a2df886-cb1c-43fa-87ef-067cc88830b7-tls-certs\") pod \"kube-auth-proxy-5f67c6995-xtj92\" (UID: \"7a2df886-cb1c-43fa-87ef-067cc88830b7\") " pod="openshift-ingress/kube-auth-proxy-5f67c6995-xtj92" Jul 07 14:30:10.766752 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.766718 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwghs\" (UniqueName: \"kubernetes.io/projected/7a2df886-cb1c-43fa-87ef-067cc88830b7-kube-api-access-qwghs\") pod \"kube-auth-proxy-5f67c6995-xtj92\" (UID: \"7a2df886-cb1c-43fa-87ef-067cc88830b7\") " pod="openshift-ingress/kube-auth-proxy-5f67c6995-xtj92" Jul 07 14:30:10.867842 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:10.867796 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f67c6995-xtj92" Jul 07 14:30:11.005443 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:11.005410 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f67c6995-xtj92"] Jul 07 14:30:11.007915 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:30:11.007889 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2df886_cb1c_43fa_87ef_067cc88830b7.slice/crio-85fe95c064f26ba47de195c8c71da1db76dcfc2f72e78412df4944fcc3405093 WatchSource:0}: Error finding container 85fe95c064f26ba47de195c8c71da1db76dcfc2f72e78412df4944fcc3405093: Status 404 returned error can't find the container with id 85fe95c064f26ba47de195c8c71da1db76dcfc2f72e78412df4944fcc3405093 Jul 07 14:30:11.149528 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:11.149488 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f67c6995-xtj92" event={"ID":"7a2df886-cb1c-43fa-87ef-067cc88830b7","Type":"ContainerStarted","Data":"85fe95c064f26ba47de195c8c71da1db76dcfc2f72e78412df4944fcc3405093"} Jul 07 14:30:12.270729 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:12.270696 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:12.371303 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:12.371247 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:13.105656 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:13.105617 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:14.166253 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:14.166216 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f67c6995-xtj92" event={"ID":"7a2df886-cb1c-43fa-87ef-067cc88830b7","Type":"ContainerStarted","Data":"6c38db02abd7dce3eb23f5ba2929d0ef40e203c48ac775849376b1555e0946ff"} Jul 07 14:30:14.189439 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:14.189361 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5f67c6995-xtj92" podStartSLOduration=1.2044243749999999 podStartE2EDuration="4.189345826s" totalImagesPullingTime="2.984921451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:30:14.18886902 +0000 UTC m=+337.998416510" watchObservedRunningTime="2026-07-07 14:30:14.189345826 +0000 UTC m=+337.998893316" Jul 07 14:30:14.923854 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:14.923819 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:16.097126 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:16.097089 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:16.232227 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:16.232189 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:17.125362 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:17.125327 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:17.297169 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:17.297133 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:17.381631 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:17.381547 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:17.797427 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:17.797396 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:17.916247 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:17.916217 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:18.200046 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:18.200009 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:18.354156 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:18.354125 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:18.878729 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:18.878680 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:18.982921 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:18.982886 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:19.068026 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:19.067979 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:19.142684 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:19.142364 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5b9884d45d-lxtjt" Jul 07 14:30:28.615820 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:28.615734 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:28.688572 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:28.688542 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:31.016780 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:31.016744 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:33.048620 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:33.048585 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:33.227459 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:33.227429 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:36.650765 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:36.650731 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:36.715240 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:36.715213 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:55.139429 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:55.139396 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:30:55.207046 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:30:55.207010 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:06.222107 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:06.222048 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:06.319559 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:06.319527 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:06.417888 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:06.417854 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:07.139556 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:07.139523 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:07.351730 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:07.351695 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:35.058994 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:35.058960 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:39.300453 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:39.300344 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:41.700665 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:41.700630 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:41.929162 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:41.929126 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:43.111306 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:43.111271 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:46.263137 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:46.263100 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:46.652816 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:46.652787 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:46.914246 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:46.914160 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:47.030105 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:47.030072 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:49.694560 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:49.694530 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:51.099306 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:51.099271 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:51.203898 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:51.203863 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:54.877614 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:54.877583 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:54.984076 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:54.984042 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:56.913726 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:56.913692 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-6ngls"] Jul 07 14:31:56.917134 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:56.917117 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6ngls" Jul 07 14:31:56.919855 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:56.919827 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 07 14:31:56.920030 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:56.919879 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 07 14:31:56.920030 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:56.919823 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 07 14:31:56.920898 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:56.920882 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 07 14:31:56.920955 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:56.920888 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4x5zn\"" Jul 07 14:31:56.927242 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:56.927218 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-6ngls"] Jul 07 14:31:56.934479 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:56.934449 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:57.098054 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:57.098011 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcw27\" (UniqueName: \"kubernetes.io/projected/56a9a81d-3332-4687-ba1c-c32bb4f8e4bd-kube-api-access-lcw27\") pod \"kuadrant-console-plugin-66b5774985-6ngls\" (UID: \"56a9a81d-3332-4687-ba1c-c32bb4f8e4bd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6ngls" Jul 07 14:31:57.098245 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:57.098074 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/56a9a81d-3332-4687-ba1c-c32bb4f8e4bd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-6ngls\" (UID: \"56a9a81d-3332-4687-ba1c-c32bb4f8e4bd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6ngls" Jul 07 14:31:57.098245 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:57.098152 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/56a9a81d-3332-4687-ba1c-c32bb4f8e4bd-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-6ngls\" (UID: \"56a9a81d-3332-4687-ba1c-c32bb4f8e4bd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6ngls" Jul 07 14:31:57.195465 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:57.195356 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:57.199475 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:57.199453 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/56a9a81d-3332-4687-ba1c-c32bb4f8e4bd-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-6ngls\" (UID: \"56a9a81d-3332-4687-ba1c-c32bb4f8e4bd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6ngls" Jul 07 14:31:57.199568 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:57.199519 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcw27\" (UniqueName: \"kubernetes.io/projected/56a9a81d-3332-4687-ba1c-c32bb4f8e4bd-kube-api-access-lcw27\") pod \"kuadrant-console-plugin-66b5774985-6ngls\" (UID: \"56a9a81d-3332-4687-ba1c-c32bb4f8e4bd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6ngls" Jul 07 14:31:57.199568 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:57.199555 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/56a9a81d-3332-4687-ba1c-c32bb4f8e4bd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-6ngls\" (UID: \"56a9a81d-3332-4687-ba1c-c32bb4f8e4bd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6ngls" Jul 07 14:31:57.200100 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:57.200076 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/56a9a81d-3332-4687-ba1c-c32bb4f8e4bd-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-6ngls\" (UID: \"56a9a81d-3332-4687-ba1c-c32bb4f8e4bd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6ngls" Jul 07 14:31:57.201934 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:57.201916 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/56a9a81d-3332-4687-ba1c-c32bb4f8e4bd-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-6ngls\" (UID: \"56a9a81d-3332-4687-ba1c-c32bb4f8e4bd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6ngls" Jul 07 14:31:57.210481 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:57.210457 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcw27\" (UniqueName: \"kubernetes.io/projected/56a9a81d-3332-4687-ba1c-c32bb4f8e4bd-kube-api-access-lcw27\") pod \"kuadrant-console-plugin-66b5774985-6ngls\" (UID: \"56a9a81d-3332-4687-ba1c-c32bb4f8e4bd\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6ngls" Jul 07 14:31:57.227158 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:57.227132 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6ngls" Jul 07 14:31:57.364936 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:57.364895 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-6ngls"] Jul 07 14:31:57.369775 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:31:57.369733 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod56a9a81d_3332_4687_ba1c_c32bb4f8e4bd.slice/crio-99b6977b6751bb2bc20421dcba479188fd6402bbd8de56f99d83fd1b2cdf55af WatchSource:0}: Error finding container 99b6977b6751bb2bc20421dcba479188fd6402bbd8de56f99d83fd1b2cdf55af: Status 404 returned error can't find the container with id 99b6977b6751bb2bc20421dcba479188fd6402bbd8de56f99d83fd1b2cdf55af Jul 07 14:31:57.626057 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:57.625974 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6ngls" event={"ID":"56a9a81d-3332-4687-ba1c-c32bb4f8e4bd","Type":"ContainerStarted","Data":"99b6977b6751bb2bc20421dcba479188fd6402bbd8de56f99d83fd1b2cdf55af"} Jul 07 14:31:58.266827 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:58.266751 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:31:58.376284 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:31:58.376252 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:06.452840 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:06.452803 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m"] Jul 07 14:32:06.456160 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:06.456137 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m" Jul 07 14:32:06.458727 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:06.458702 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jbwxs\"" Jul 07 14:32:06.478232 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:06.478206 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m"] Jul 07 14:32:06.485799 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:06.485763 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsztp\" (UniqueName: \"kubernetes.io/projected/ed5b5757-6d4b-4051-891d-8b6f1133bdec-kube-api-access-nsztp\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kdk5m\" (UID: \"ed5b5757-6d4b-4051-891d-8b6f1133bdec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m" Jul 07 14:32:06.485963 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:06.485876 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed5b5757-6d4b-4051-891d-8b6f1133bdec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kdk5m\" (UID: \"ed5b5757-6d4b-4051-891d-8b6f1133bdec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m" Jul 07 14:32:06.586577 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:06.586537 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed5b5757-6d4b-4051-891d-8b6f1133bdec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kdk5m\" (UID: \"ed5b5757-6d4b-4051-891d-8b6f1133bdec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m" Jul 07 14:32:06.586763 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:06.586633 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsztp\" (UniqueName: \"kubernetes.io/projected/ed5b5757-6d4b-4051-891d-8b6f1133bdec-kube-api-access-nsztp\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kdk5m\" (UID: \"ed5b5757-6d4b-4051-891d-8b6f1133bdec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m" Jul 07 14:32:06.587001 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:06.586968 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed5b5757-6d4b-4051-891d-8b6f1133bdec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kdk5m\" (UID: \"ed5b5757-6d4b-4051-891d-8b6f1133bdec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m" Jul 07 14:32:06.598426 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:06.598399 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsztp\" (UniqueName: \"kubernetes.io/projected/ed5b5757-6d4b-4051-891d-8b6f1133bdec-kube-api-access-nsztp\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-kdk5m\" (UID: \"ed5b5757-6d4b-4051-891d-8b6f1133bdec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m" Jul 07 14:32:06.734588 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:06.734505 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:06.766759 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:06.766728 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m" Jul 07 14:32:06.838733 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:06.838704 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:06.914324 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:06.914207 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m"] Jul 07 14:32:07.309936 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:07.309900 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m"] Jul 07 14:32:07.320444 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:07.320405 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m"] Jul 07 14:32:07.342899 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:07.342868 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl"] Jul 07 14:32:07.350239 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:07.350210 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl" Jul 07 14:32:07.369150 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:07.369066 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl"] Jul 07 14:32:07.393553 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:07.393506 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fxb5\" (UniqueName: \"kubernetes.io/projected/4936dc3d-f9fd-404a-b4ab-93557ba3e777-kube-api-access-8fxb5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-nzkvl\" (UID: \"4936dc3d-f9fd-404a-b4ab-93557ba3e777\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl" Jul 07 14:32:07.393720 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:07.393564 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4936dc3d-f9fd-404a-b4ab-93557ba3e777-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-nzkvl\" (UID: \"4936dc3d-f9fd-404a-b4ab-93557ba3e777\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl" Jul 07 14:32:07.494843 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:07.494807 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fxb5\" (UniqueName: \"kubernetes.io/projected/4936dc3d-f9fd-404a-b4ab-93557ba3e777-kube-api-access-8fxb5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-nzkvl\" (UID: \"4936dc3d-f9fd-404a-b4ab-93557ba3e777\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl" Jul 07 14:32:07.495316 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:07.494866 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4936dc3d-f9fd-404a-b4ab-93557ba3e777-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-nzkvl\" (UID: \"4936dc3d-f9fd-404a-b4ab-93557ba3e777\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl" Jul 07 14:32:07.495369 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:07.495318 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4936dc3d-f9fd-404a-b4ab-93557ba3e777-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-nzkvl\" (UID: \"4936dc3d-f9fd-404a-b4ab-93557ba3e777\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl" Jul 07 14:32:07.508523 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:07.508491 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fxb5\" (UniqueName: \"kubernetes.io/projected/4936dc3d-f9fd-404a-b4ab-93557ba3e777-kube-api-access-8fxb5\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-nzkvl\" (UID: \"4936dc3d-f9fd-404a-b4ab-93557ba3e777\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl" Jul 07 14:32:07.666621 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:07.666582 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl" Jul 07 14:32:09.341232 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:09.341194 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:09.452830 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:09.452797 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:12.169992 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:12.169951 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:12.267869 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:12.267824 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:14.894601 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:14.894561 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:14.995893 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:14.995854 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:15.691292 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:15.691262 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:15.791622 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:15.791589 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:21.392242 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:32:21.392204 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5b5757_6d4b_4051_891d_8b6f1133bdec.slice/crio-1a971e789dcbde0c66ca0fe01cb94e05b5f54455808567b8c4d7a7d618d470cd WatchSource:0}: Error finding container 1a971e789dcbde0c66ca0fe01cb94e05b5f54455808567b8c4d7a7d618d470cd: Status 404 returned error can't find the container with id 1a971e789dcbde0c66ca0fe01cb94e05b5f54455808567b8c4d7a7d618d470cd Jul 07 14:32:21.534654 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:21.534625 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl"] Jul 07 14:32:21.538398 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:32:21.538353 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4936dc3d_f9fd_404a_b4ab_93557ba3e777.slice/crio-0821a5170ab35e53297539fabb8005c485f5732a942f1357b477db33205b2332 WatchSource:0}: Error finding container 0821a5170ab35e53297539fabb8005c485f5732a942f1357b477db33205b2332: Status 404 returned error can't find the container with id 0821a5170ab35e53297539fabb8005c485f5732a942f1357b477db33205b2332 Jul 07 14:32:21.757552 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:21.757514 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6ngls" event={"ID":"56a9a81d-3332-4687-ba1c-c32bb4f8e4bd","Type":"ContainerStarted","Data":"4961c055704d486179bb63f2713e9e43dc1209ddce4b8f93721f50dbe5d83829"} Jul 07 14:32:21.758811 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:21.758770 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl" event={"ID":"4936dc3d-f9fd-404a-b4ab-93557ba3e777","Type":"ContainerStarted","Data":"0821a5170ab35e53297539fabb8005c485f5732a942f1357b477db33205b2332"} Jul 07 14:32:21.776985 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:21.776930 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-6ngls" podStartSLOduration=1.675600625 podStartE2EDuration="25.776912745s" totalImagesPullingTime="24.10131212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:31:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:32:21.775716599 +0000 UTC m=+465.585264089" watchObservedRunningTime="2026-07-07 14:32:21.776912745 +0000 UTC m=+465.586460235" Jul 07 14:32:24.568312 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:24.568279 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:24.674671 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:24.674640 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:27.159469 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:27.159436 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:27.263109 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:27.263076 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:27.789971 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:27.789929 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m" podUID="ed5b5757-6d4b-4051-891d-8b6f1133bdec" containerName="manager" containerID="cri-o://f35e7830fd3c09b7d9f18cae4facb8e497fcfea182cb5e43852e1bf56c8f71aa" gracePeriod=2 Jul 07 14:32:27.791674 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:27.791639 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl" event={"ID":"4936dc3d-f9fd-404a-b4ab-93557ba3e777","Type":"ContainerStarted","Data":"2d8e2b6a3668308cd23294fc0a430e84e022fd6dae648cbc244ab02bf6104053"} Jul 07 14:32:27.791830 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:27.791735 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl" Jul 07 14:32:27.792305 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:32:27.792278 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kdk5m\" is forbidden: User \"system:node:ip-10-0-128-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-34.ec2.internal' and this object" podUID="ed5b5757-6d4b-4051-891d-8b6f1133bdec" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m" Jul 07 14:32:27.794650 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:32:27.794534 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kdk5m\" is forbidden: User \"system:node:ip-10-0-128-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-34.ec2.internal' and this object" podUID="ed5b5757-6d4b-4051-891d-8b6f1133bdec" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m" Jul 07 14:32:27.823777 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:27.823728 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl" podStartSLOduration=15.251709587 podStartE2EDuration="20.823710198s" totalImagesPullingTime="5.572000611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:32:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:32:27.821302862 +0000 UTC m=+471.630850354" watchObservedRunningTime="2026-07-07 14:32:27.823710198 +0000 UTC m=+471.633257687" Jul 07 14:32:28.040225 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:28.040158 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m" Jul 07 14:32:28.043053 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:32:28.043026 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kdk5m\" is forbidden: User \"system:node:ip-10-0-128-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-34.ec2.internal' and this object" podUID="ed5b5757-6d4b-4051-891d-8b6f1133bdec" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m" Jul 07 14:32:28.203771 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:28.203731 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsztp\" (UniqueName: \"kubernetes.io/projected/ed5b5757-6d4b-4051-891d-8b6f1133bdec-kube-api-access-nsztp\") pod \"ed5b5757-6d4b-4051-891d-8b6f1133bdec\" (UID: \"ed5b5757-6d4b-4051-891d-8b6f1133bdec\") " Jul 07 14:32:28.204150 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:28.203791 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed5b5757-6d4b-4051-891d-8b6f1133bdec-extensions-socket-volume\") pod \"ed5b5757-6d4b-4051-891d-8b6f1133bdec\" (UID: \"ed5b5757-6d4b-4051-891d-8b6f1133bdec\") " Jul 07 14:32:28.204150 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:28.204078 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5b5757-6d4b-4051-891d-8b6f1133bdec-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ed5b5757-6d4b-4051-891d-8b6f1133bdec" (UID: "ed5b5757-6d4b-4051-891d-8b6f1133bdec"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:32:28.206080 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:28.206060 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5b5757-6d4b-4051-891d-8b6f1133bdec-kube-api-access-nsztp" (OuterVolumeSpecName: "kube-api-access-nsztp") pod "ed5b5757-6d4b-4051-891d-8b6f1133bdec" (UID: "ed5b5757-6d4b-4051-891d-8b6f1133bdec"). InnerVolumeSpecName "kube-api-access-nsztp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:32:28.304910 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:28.304830 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsztp\" (UniqueName: \"kubernetes.io/projected/ed5b5757-6d4b-4051-891d-8b6f1133bdec-kube-api-access-nsztp\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:32:28.304910 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:28.304858 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed5b5757-6d4b-4051-891d-8b6f1133bdec-extensions-socket-volume\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:32:28.706521 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:28.706490 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed5b5757-6d4b-4051-891d-8b6f1133bdec" path="/var/lib/kubelet/pods/ed5b5757-6d4b-4051-891d-8b6f1133bdec/volumes" Jul 07 14:32:28.798073 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:28.798037 2562 generic.go:356] "Generic (PLEG): container finished" podID="ed5b5757-6d4b-4051-891d-8b6f1133bdec" containerID="f35e7830fd3c09b7d9f18cae4facb8e497fcfea182cb5e43852e1bf56c8f71aa" exitCode=2 Jul 07 14:32:28.798261 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:28.798089 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m" Jul 07 14:32:28.798261 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:28.798140 2562 scope.go:117] "RemoveContainer" containerID="f35e7830fd3c09b7d9f18cae4facb8e497fcfea182cb5e43852e1bf56c8f71aa" Jul 07 14:32:28.800824 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:32:28.800798 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kdk5m\" is forbidden: User \"system:node:ip-10-0-128-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-34.ec2.internal' and this object" podUID="ed5b5757-6d4b-4051-891d-8b6f1133bdec" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m" Jul 07 14:32:28.802905 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:32:28.802876 2562 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-kdk5m\" is forbidden: User \"system:node:ip-10-0-128-34.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-34.ec2.internal' and this object" podUID="ed5b5757-6d4b-4051-891d-8b6f1133bdec" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-kdk5m" Jul 07 14:32:28.806697 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:28.806682 2562 scope.go:117] "RemoveContainer" containerID="f35e7830fd3c09b7d9f18cae4facb8e497fcfea182cb5e43852e1bf56c8f71aa" Jul 07 14:32:28.806968 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:32:28.806949 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35e7830fd3c09b7d9f18cae4facb8e497fcfea182cb5e43852e1bf56c8f71aa\": container with ID starting with f35e7830fd3c09b7d9f18cae4facb8e497fcfea182cb5e43852e1bf56c8f71aa not found: ID does not exist" containerID="f35e7830fd3c09b7d9f18cae4facb8e497fcfea182cb5e43852e1bf56c8f71aa" Jul 07 14:32:28.807030 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:28.806977 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35e7830fd3c09b7d9f18cae4facb8e497fcfea182cb5e43852e1bf56c8f71aa"} err="failed to get container status \"f35e7830fd3c09b7d9f18cae4facb8e497fcfea182cb5e43852e1bf56c8f71aa\": rpc error: code = NotFound desc = could not find container \"f35e7830fd3c09b7d9f18cae4facb8e497fcfea182cb5e43852e1bf56c8f71aa\": container with ID starting with f35e7830fd3c09b7d9f18cae4facb8e497fcfea182cb5e43852e1bf56c8f71aa not found: ID does not exist" Jul 07 14:32:30.753925 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:30.753892 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:30.852246 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:30.852200 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:33.946461 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:33.946426 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:34.046391 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:34.046328 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:38.800506 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:38.800471 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl" Jul 07 14:32:44.153851 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:44.153807 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:44.254674 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:44.254633 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:44.359731 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:44.359673 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:44.432413 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:44.432317 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:49.283270 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:49.283239 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:49.348785 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:49.348751 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:51.708078 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:51.708039 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl"] Jul 07 14:32:51.708506 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:51.708242 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl" podUID="4936dc3d-f9fd-404a-b4ab-93557ba3e777" containerName="manager" containerID="cri-o://2d8e2b6a3668308cd23294fc0a430e84e022fd6dae648cbc244ab02bf6104053" gracePeriod=10 Jul 07 14:32:51.914250 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:51.914220 2562 generic.go:356] "Generic (PLEG): container finished" podID="4936dc3d-f9fd-404a-b4ab-93557ba3e777" containerID="2d8e2b6a3668308cd23294fc0a430e84e022fd6dae648cbc244ab02bf6104053" exitCode=0 Jul 07 14:32:51.914441 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:51.914275 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl" event={"ID":"4936dc3d-f9fd-404a-b4ab-93557ba3e777","Type":"ContainerDied","Data":"2d8e2b6a3668308cd23294fc0a430e84e022fd6dae648cbc244ab02bf6104053"} Jul 07 14:32:51.955761 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:51.955736 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl" Jul 07 14:32:52.020438 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:52.020317 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fxb5\" (UniqueName: \"kubernetes.io/projected/4936dc3d-f9fd-404a-b4ab-93557ba3e777-kube-api-access-8fxb5\") pod \"4936dc3d-f9fd-404a-b4ab-93557ba3e777\" (UID: \"4936dc3d-f9fd-404a-b4ab-93557ba3e777\") " Jul 07 14:32:52.020438 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:52.020368 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4936dc3d-f9fd-404a-b4ab-93557ba3e777-extensions-socket-volume\") pod \"4936dc3d-f9fd-404a-b4ab-93557ba3e777\" (UID: \"4936dc3d-f9fd-404a-b4ab-93557ba3e777\") " Jul 07 14:32:52.020771 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:52.020749 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4936dc3d-f9fd-404a-b4ab-93557ba3e777-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4936dc3d-f9fd-404a-b4ab-93557ba3e777" (UID: "4936dc3d-f9fd-404a-b4ab-93557ba3e777"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:32:52.022410 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:52.022390 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4936dc3d-f9fd-404a-b4ab-93557ba3e777-kube-api-access-8fxb5" (OuterVolumeSpecName: "kube-api-access-8fxb5") pod "4936dc3d-f9fd-404a-b4ab-93557ba3e777" (UID: "4936dc3d-f9fd-404a-b4ab-93557ba3e777"). InnerVolumeSpecName "kube-api-access-8fxb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:32:52.121171 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:52.121136 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fxb5\" (UniqueName: \"kubernetes.io/projected/4936dc3d-f9fd-404a-b4ab-93557ba3e777-kube-api-access-8fxb5\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:32:52.121171 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:52.121165 2562 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4936dc3d-f9fd-404a-b4ab-93557ba3e777-extensions-socket-volume\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:32:52.920340 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:52.920309 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl" Jul 07 14:32:52.920810 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:52.920341 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl" event={"ID":"4936dc3d-f9fd-404a-b4ab-93557ba3e777","Type":"ContainerDied","Data":"0821a5170ab35e53297539fabb8005c485f5732a942f1357b477db33205b2332"} Jul 07 14:32:52.920810 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:52.920422 2562 scope.go:117] "RemoveContainer" containerID="2d8e2b6a3668308cd23294fc0a430e84e022fd6dae648cbc244ab02bf6104053" Jul 07 14:32:52.948890 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:52.948860 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl"] Jul 07 14:32:52.955184 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:52.955157 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-nzkvl"] Jul 07 14:32:54.707191 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:54.707162 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4936dc3d-f9fd-404a-b4ab-93557ba3e777" path="/var/lib/kubelet/pods/4936dc3d-f9fd-404a-b4ab-93557ba3e777/volumes" Jul 07 14:32:55.400182 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:55.400146 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:55.498595 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:55.498559 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:57.969173 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:57.969133 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:57.972344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:57.972319 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wsf5j"] Jul 07 14:32:57.972669 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:57.972656 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4936dc3d-f9fd-404a-b4ab-93557ba3e777" containerName="manager" Jul 07 14:32:57.972710 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:57.972671 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4936dc3d-f9fd-404a-b4ab-93557ba3e777" containerName="manager" Jul 07 14:32:57.972710 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:57.972698 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed5b5757-6d4b-4051-891d-8b6f1133bdec" containerName="manager" Jul 07 14:32:57.972710 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:57.972703 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed5b5757-6d4b-4051-891d-8b6f1133bdec" containerName="manager" Jul 07 14:32:57.972797 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:57.972756 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4936dc3d-f9fd-404a-b4ab-93557ba3e777" containerName="manager" Jul 07 14:32:57.972797 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:57.972764 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed5b5757-6d4b-4051-891d-8b6f1133bdec" containerName="manager" Jul 07 14:32:58.058732 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.058697 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wsf5j"] Jul 07 14:32:58.058882 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.058823 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wsf5j" Jul 07 14:32:58.061527 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.061498 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 07 14:32:58.067144 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.066636 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wsf5j"] Jul 07 14:32:58.070771 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.070747 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:58.152527 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.152491 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:58.178742 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.178709 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b2351b80-5782-4d0b-bff5-55989faeb666-config-file\") pod \"limitador-limitador-64c94fd64f-wsf5j\" (UID: \"b2351b80-5782-4d0b-bff5-55989faeb666\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wsf5j" Jul 07 14:32:58.178925 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.178763 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc5t7\" (UniqueName: \"kubernetes.io/projected/b2351b80-5782-4d0b-bff5-55989faeb666-kube-api-access-lc5t7\") pod \"limitador-limitador-64c94fd64f-wsf5j\" (UID: \"b2351b80-5782-4d0b-bff5-55989faeb666\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wsf5j" Jul 07 14:32:58.280244 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.280159 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b2351b80-5782-4d0b-bff5-55989faeb666-config-file\") pod \"limitador-limitador-64c94fd64f-wsf5j\" (UID: \"b2351b80-5782-4d0b-bff5-55989faeb666\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wsf5j" Jul 07 14:32:58.280244 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.280213 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc5t7\" (UniqueName: \"kubernetes.io/projected/b2351b80-5782-4d0b-bff5-55989faeb666-kube-api-access-lc5t7\") pod \"limitador-limitador-64c94fd64f-wsf5j\" (UID: \"b2351b80-5782-4d0b-bff5-55989faeb666\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wsf5j" Jul 07 14:32:58.280832 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.280810 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b2351b80-5782-4d0b-bff5-55989faeb666-config-file\") pod \"limitador-limitador-64c94fd64f-wsf5j\" (UID: \"b2351b80-5782-4d0b-bff5-55989faeb666\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wsf5j" Jul 07 14:32:58.291168 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.291138 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc5t7\" (UniqueName: \"kubernetes.io/projected/b2351b80-5782-4d0b-bff5-55989faeb666-kube-api-access-lc5t7\") pod \"limitador-limitador-64c94fd64f-wsf5j\" (UID: \"b2351b80-5782-4d0b-bff5-55989faeb666\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-wsf5j" Jul 07 14:32:58.371782 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.371742 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wsf5j" Jul 07 14:32:58.512231 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.512203 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wsf5j"] Jul 07 14:32:58.514382 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:32:58.514345 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2351b80_5782_4d0b_bff5_55989faeb666.slice/crio-bc0e58db92244e3c24cc38b6375d93b84e0519da076e06bbdca4b33c744035f0 WatchSource:0}: Error finding container bc0e58db92244e3c24cc38b6375d93b84e0519da076e06bbdca4b33c744035f0: Status 404 returned error can't find the container with id bc0e58db92244e3c24cc38b6375d93b84e0519da076e06bbdca4b33c744035f0 Jul 07 14:32:58.883262 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.883221 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-cq9xh"] Jul 07 14:32:58.887992 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.887970 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cq9xh" Jul 07 14:32:58.891190 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.891171 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4x47t\"" Jul 07 14:32:58.899477 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.899456 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-cq9xh"] Jul 07 14:32:58.906583 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.906565 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:58.951403 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.951348 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wsf5j" event={"ID":"b2351b80-5782-4d0b-bff5-55989faeb666","Type":"ContainerStarted","Data":"bc0e58db92244e3c24cc38b6375d93b84e0519da076e06bbdca4b33c744035f0"} Jul 07 14:32:58.987712 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:58.987670 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfl8p\" (UniqueName: \"kubernetes.io/projected/25fccd52-c709-4201-9246-a7c7413d9d25-kube-api-access-dfl8p\") pod \"authorino-58cd955675-cq9xh\" (UID: \"25fccd52-c709-4201-9246-a7c7413d9d25\") " pod="kuadrant-system/authorino-58cd955675-cq9xh" Jul 07 14:32:59.004414 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:59.004363 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:32:59.088684 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:59.088651 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfl8p\" (UniqueName: \"kubernetes.io/projected/25fccd52-c709-4201-9246-a7c7413d9d25-kube-api-access-dfl8p\") pod \"authorino-58cd955675-cq9xh\" (UID: \"25fccd52-c709-4201-9246-a7c7413d9d25\") " pod="kuadrant-system/authorino-58cd955675-cq9xh" Jul 07 14:32:59.104858 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:59.104827 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfl8p\" (UniqueName: \"kubernetes.io/projected/25fccd52-c709-4201-9246-a7c7413d9d25-kube-api-access-dfl8p\") pod \"authorino-58cd955675-cq9xh\" (UID: \"25fccd52-c709-4201-9246-a7c7413d9d25\") " pod="kuadrant-system/authorino-58cd955675-cq9xh" Jul 07 14:32:59.197553 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:59.197519 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cq9xh" Jul 07 14:32:59.372924 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:59.372861 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-cq9xh"] Jul 07 14:32:59.377078 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:32:59.377004 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25fccd52_c709_4201_9246_a7c7413d9d25.slice/crio-9cfa17fd00f3f60372944f60454261d99c0e769762c91cf49fbe6373e93cdb3c WatchSource:0}: Error finding container 9cfa17fd00f3f60372944f60454261d99c0e769762c91cf49fbe6373e93cdb3c: Status 404 returned error can't find the container with id 9cfa17fd00f3f60372944f60454261d99c0e769762c91cf49fbe6373e93cdb3c Jul 07 14:32:59.957450 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:32:59.957414 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cq9xh" event={"ID":"25fccd52-c709-4201-9246-a7c7413d9d25","Type":"ContainerStarted","Data":"9cfa17fd00f3f60372944f60454261d99c0e769762c91cf49fbe6373e93cdb3c"} Jul 07 14:33:03.872882 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:03.872843 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-cq9xh"] Jul 07 14:33:03.981474 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:03.981439 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wsf5j" event={"ID":"b2351b80-5782-4d0b-bff5-55989faeb666","Type":"ContainerStarted","Data":"9e0cf82993d06358a1dac90fdb442a34fd78a53c69bb4df24cd67479526b4946"} Jul 07 14:33:03.981650 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:03.981527 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-wsf5j" Jul 07 14:33:03.983025 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:03.982998 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cq9xh" event={"ID":"25fccd52-c709-4201-9246-a7c7413d9d25","Type":"ContainerStarted","Data":"a27f97f5bf2b605b7f1f5e819b8d5e5767ad29781ae611e75cee656528cf07d0"} Jul 07 14:33:04.001928 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:04.001869 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-wsf5j" podStartSLOduration=2.249569976 podStartE2EDuration="7.001849206s" totalImagesPullingTime="4.75227923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:33:04.001129008 +0000 UTC m=+507.810676526" watchObservedRunningTime="2026-07-07 14:33:04.001849206 +0000 UTC m=+507.811396697" Jul 07 14:33:04.019903 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:04.019852 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-cq9xh" podStartSLOduration=2.070719509 podStartE2EDuration="6.019836586s" totalImagesPullingTime="3.949117077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:33:04.018927383 +0000 UTC m=+507.828474874" watchObservedRunningTime="2026-07-07 14:33:04.019836586 +0000 UTC m=+507.829384075" Jul 07 14:33:04.987334 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:04.987290 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-cq9xh" podUID="25fccd52-c709-4201-9246-a7c7413d9d25" containerName="authorino" containerID="cri-o://a27f97f5bf2b605b7f1f5e819b8d5e5767ad29781ae611e75cee656528cf07d0" gracePeriod=30 Jul 07 14:33:05.239246 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:05.239183 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cq9xh" Jul 07 14:33:05.351205 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:05.351171 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dfl8p\" (UniqueName: \"kubernetes.io/projected/25fccd52-c709-4201-9246-a7c7413d9d25-kube-api-access-dfl8p\") pod \"25fccd52-c709-4201-9246-a7c7413d9d25\" (UID: \"25fccd52-c709-4201-9246-a7c7413d9d25\") " Jul 07 14:33:05.353167 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:05.353142 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25fccd52-c709-4201-9246-a7c7413d9d25-kube-api-access-dfl8p" (OuterVolumeSpecName: "kube-api-access-dfl8p") pod "25fccd52-c709-4201-9246-a7c7413d9d25" (UID: "25fccd52-c709-4201-9246-a7c7413d9d25"). InnerVolumeSpecName "kube-api-access-dfl8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:33:05.452574 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:05.452533 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfl8p\" (UniqueName: \"kubernetes.io/projected/25fccd52-c709-4201-9246-a7c7413d9d25-kube-api-access-dfl8p\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:33:05.992649 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:05.992614 2562 generic.go:356] "Generic (PLEG): container finished" podID="25fccd52-c709-4201-9246-a7c7413d9d25" containerID="a27f97f5bf2b605b7f1f5e819b8d5e5767ad29781ae611e75cee656528cf07d0" exitCode=0 Jul 07 14:33:05.993150 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:05.992655 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cq9xh" Jul 07 14:33:05.993150 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:05.992681 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cq9xh" event={"ID":"25fccd52-c709-4201-9246-a7c7413d9d25","Type":"ContainerDied","Data":"a27f97f5bf2b605b7f1f5e819b8d5e5767ad29781ae611e75cee656528cf07d0"} Jul 07 14:33:05.993150 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:05.992711 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cq9xh" event={"ID":"25fccd52-c709-4201-9246-a7c7413d9d25","Type":"ContainerDied","Data":"9cfa17fd00f3f60372944f60454261d99c0e769762c91cf49fbe6373e93cdb3c"} Jul 07 14:33:05.993150 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:05.992731 2562 scope.go:117] "RemoveContainer" containerID="a27f97f5bf2b605b7f1f5e819b8d5e5767ad29781ae611e75cee656528cf07d0" Jul 07 14:33:06.001479 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:06.001460 2562 scope.go:117] "RemoveContainer" containerID="a27f97f5bf2b605b7f1f5e819b8d5e5767ad29781ae611e75cee656528cf07d0" Jul 07 14:33:06.001738 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:33:06.001721 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27f97f5bf2b605b7f1f5e819b8d5e5767ad29781ae611e75cee656528cf07d0\": container with ID starting with a27f97f5bf2b605b7f1f5e819b8d5e5767ad29781ae611e75cee656528cf07d0 not found: ID does not exist" containerID="a27f97f5bf2b605b7f1f5e819b8d5e5767ad29781ae611e75cee656528cf07d0" Jul 07 14:33:06.001781 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:06.001748 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27f97f5bf2b605b7f1f5e819b8d5e5767ad29781ae611e75cee656528cf07d0"} err="failed to get container status \"a27f97f5bf2b605b7f1f5e819b8d5e5767ad29781ae611e75cee656528cf07d0\": rpc error: code = NotFound desc = could not find container \"a27f97f5bf2b605b7f1f5e819b8d5e5767ad29781ae611e75cee656528cf07d0\": container with ID starting with a27f97f5bf2b605b7f1f5e819b8d5e5767ad29781ae611e75cee656528cf07d0 not found: ID does not exist" Jul 07 14:33:06.015154 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:06.015124 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-cq9xh"] Jul 07 14:33:06.020388 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:06.020350 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-cq9xh"] Jul 07 14:33:06.711173 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:06.711137 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25fccd52-c709-4201-9246-a7c7413d9d25" path="/var/lib/kubelet/pods/25fccd52-c709-4201-9246-a7c7413d9d25/volumes" Jul 07 14:33:12.806546 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:12.806514 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wsf5j"] Jul 07 14:33:12.806970 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:12.806693 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-wsf5j" podUID="b2351b80-5782-4d0b-bff5-55989faeb666" containerName="limitador" containerID="cri-o://9e0cf82993d06358a1dac90fdb442a34fd78a53c69bb4df24cd67479526b4946" gracePeriod=30 Jul 07 14:33:12.807470 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:12.807385 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-wsf5j" Jul 07 14:33:13.360106 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:13.360081 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wsf5j" Jul 07 14:33:13.526875 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:13.526822 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc5t7\" (UniqueName: \"kubernetes.io/projected/b2351b80-5782-4d0b-bff5-55989faeb666-kube-api-access-lc5t7\") pod \"b2351b80-5782-4d0b-bff5-55989faeb666\" (UID: \"b2351b80-5782-4d0b-bff5-55989faeb666\") " Jul 07 14:33:13.527068 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:13.526900 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b2351b80-5782-4d0b-bff5-55989faeb666-config-file\") pod \"b2351b80-5782-4d0b-bff5-55989faeb666\" (UID: \"b2351b80-5782-4d0b-bff5-55989faeb666\") " Jul 07 14:33:13.527258 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:13.527232 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2351b80-5782-4d0b-bff5-55989faeb666-config-file" (OuterVolumeSpecName: "config-file") pod "b2351b80-5782-4d0b-bff5-55989faeb666" (UID: "b2351b80-5782-4d0b-bff5-55989faeb666"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 14:33:13.528947 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:13.528925 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2351b80-5782-4d0b-bff5-55989faeb666-kube-api-access-lc5t7" (OuterVolumeSpecName: "kube-api-access-lc5t7") pod "b2351b80-5782-4d0b-bff5-55989faeb666" (UID: "b2351b80-5782-4d0b-bff5-55989faeb666"). InnerVolumeSpecName "kube-api-access-lc5t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:33:13.628253 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:13.628217 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc5t7\" (UniqueName: \"kubernetes.io/projected/b2351b80-5782-4d0b-bff5-55989faeb666-kube-api-access-lc5t7\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:33:13.628253 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:13.628246 2562 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b2351b80-5782-4d0b-bff5-55989faeb666-config-file\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:33:14.032234 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.032201 2562 generic.go:356] "Generic (PLEG): container finished" podID="b2351b80-5782-4d0b-bff5-55989faeb666" containerID="9e0cf82993d06358a1dac90fdb442a34fd78a53c69bb4df24cd67479526b4946" exitCode=0 Jul 07 14:33:14.032756 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.032245 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-wsf5j" Jul 07 14:33:14.032756 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.032270 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wsf5j" event={"ID":"b2351b80-5782-4d0b-bff5-55989faeb666","Type":"ContainerDied","Data":"9e0cf82993d06358a1dac90fdb442a34fd78a53c69bb4df24cd67479526b4946"} Jul 07 14:33:14.032756 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.032309 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-wsf5j" event={"ID":"b2351b80-5782-4d0b-bff5-55989faeb666","Type":"ContainerDied","Data":"bc0e58db92244e3c24cc38b6375d93b84e0519da076e06bbdca4b33c744035f0"} Jul 07 14:33:14.032756 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.032330 2562 scope.go:117] "RemoveContainer" containerID="9e0cf82993d06358a1dac90fdb442a34fd78a53c69bb4df24cd67479526b4946" Jul 07 14:33:14.041135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.041118 2562 scope.go:117] "RemoveContainer" containerID="9e0cf82993d06358a1dac90fdb442a34fd78a53c69bb4df24cd67479526b4946" Jul 07 14:33:14.041392 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:33:14.041358 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0cf82993d06358a1dac90fdb442a34fd78a53c69bb4df24cd67479526b4946\": container with ID starting with 9e0cf82993d06358a1dac90fdb442a34fd78a53c69bb4df24cd67479526b4946 not found: ID does not exist" containerID="9e0cf82993d06358a1dac90fdb442a34fd78a53c69bb4df24cd67479526b4946" Jul 07 14:33:14.041458 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.041398 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0cf82993d06358a1dac90fdb442a34fd78a53c69bb4df24cd67479526b4946"} err="failed to get container status \"9e0cf82993d06358a1dac90fdb442a34fd78a53c69bb4df24cd67479526b4946\": rpc error: code = NotFound desc = could not find container \"9e0cf82993d06358a1dac90fdb442a34fd78a53c69bb4df24cd67479526b4946\": container with ID starting with 9e0cf82993d06358a1dac90fdb442a34fd78a53c69bb4df24cd67479526b4946 not found: ID does not exist" Jul 07 14:33:14.055914 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.055885 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wsf5j"] Jul 07 14:33:14.061071 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.061048 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-wsf5j"] Jul 07 14:33:14.599232 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.599202 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-cvhmk"] Jul 07 14:33:14.599559 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.599546 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25fccd52-c709-4201-9246-a7c7413d9d25" containerName="authorino" Jul 07 14:33:14.599635 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.599560 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="25fccd52-c709-4201-9246-a7c7413d9d25" containerName="authorino" Jul 07 14:33:14.599635 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.599574 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2351b80-5782-4d0b-bff5-55989faeb666" containerName="limitador" Jul 07 14:33:14.599635 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.599581 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2351b80-5782-4d0b-bff5-55989faeb666" containerName="limitador" Jul 07 14:33:14.599785 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.599674 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2351b80-5782-4d0b-bff5-55989faeb666" containerName="limitador" Jul 07 14:33:14.599785 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.599687 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="25fccd52-c709-4201-9246-a7c7413d9d25" containerName="authorino" Jul 07 14:33:14.604069 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.604046 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-cvhmk" Jul 07 14:33:14.606510 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.606487 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lx5wr\"" Jul 07 14:33:14.606738 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.606724 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 07 14:33:14.613233 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.613203 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-cvhmk"] Jul 07 14:33:14.651783 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.651705 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:14.706407 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.706360 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2351b80-5782-4d0b-bff5-55989faeb666" path="/var/lib/kubelet/pods/b2351b80-5782-4d0b-bff5-55989faeb666/volumes" Jul 07 14:33:14.739456 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.739420 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ecf59abb-a58d-439b-901c-7beab9368363-data\") pod \"postgres-69fb4594cf-cvhmk\" (UID: \"ecf59abb-a58d-439b-901c-7beab9368363\") " pod="opendatahub/postgres-69fb4594cf-cvhmk" Jul 07 14:33:14.739621 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.739462 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk87c\" (UniqueName: \"kubernetes.io/projected/ecf59abb-a58d-439b-901c-7beab9368363-kube-api-access-rk87c\") pod \"postgres-69fb4594cf-cvhmk\" (UID: \"ecf59abb-a58d-439b-901c-7beab9368363\") " pod="opendatahub/postgres-69fb4594cf-cvhmk" Jul 07 14:33:14.840329 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.840276 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ecf59abb-a58d-439b-901c-7beab9368363-data\") pod \"postgres-69fb4594cf-cvhmk\" (UID: \"ecf59abb-a58d-439b-901c-7beab9368363\") " pod="opendatahub/postgres-69fb4594cf-cvhmk" Jul 07 14:33:14.840329 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.840327 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk87c\" (UniqueName: \"kubernetes.io/projected/ecf59abb-a58d-439b-901c-7beab9368363-kube-api-access-rk87c\") pod \"postgres-69fb4594cf-cvhmk\" (UID: \"ecf59abb-a58d-439b-901c-7beab9368363\") " pod="opendatahub/postgres-69fb4594cf-cvhmk" Jul 07 14:33:14.840790 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.840767 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/ecf59abb-a58d-439b-901c-7beab9368363-data\") pod \"postgres-69fb4594cf-cvhmk\" (UID: \"ecf59abb-a58d-439b-901c-7beab9368363\") " pod="opendatahub/postgres-69fb4594cf-cvhmk" Jul 07 14:33:14.849721 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.849650 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk87c\" (UniqueName: \"kubernetes.io/projected/ecf59abb-a58d-439b-901c-7beab9368363-kube-api-access-rk87c\") pod \"postgres-69fb4594cf-cvhmk\" (UID: \"ecf59abb-a58d-439b-901c-7beab9368363\") " pod="opendatahub/postgres-69fb4594cf-cvhmk" Jul 07 14:33:14.915387 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:14.915348 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-cvhmk" Jul 07 14:33:15.051048 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:15.050675 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-cvhmk"] Jul 07 14:33:15.053501 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:33:15.053468 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecf59abb_a58d_439b_901c_7beab9368363.slice/crio-a8db8c1e95cb5698c3060da5c7d02de9e6593f55584f3833c03d5db6bc1bbcb3 WatchSource:0}: Error finding container a8db8c1e95cb5698c3060da5c7d02de9e6593f55584f3833c03d5db6bc1bbcb3: Status 404 returned error can't find the container with id a8db8c1e95cb5698c3060da5c7d02de9e6593f55584f3833c03d5db6bc1bbcb3 Jul 07 14:33:16.044457 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:16.044418 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-cvhmk" event={"ID":"ecf59abb-a58d-439b-901c-7beab9368363","Type":"ContainerStarted","Data":"a8db8c1e95cb5698c3060da5c7d02de9e6593f55584f3833c03d5db6bc1bbcb3"} Jul 07 14:33:21.073548 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:21.073504 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-cvhmk" event={"ID":"ecf59abb-a58d-439b-901c-7beab9368363","Type":"ContainerStarted","Data":"06b70d038c589da1c6f6f2936882077df491b488c343b0094daf955f0cf03464"} Jul 07 14:33:21.074017 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:21.073570 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-cvhmk" Jul 07 14:33:21.098061 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:21.098014 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-cvhmk" podStartSLOduration=1.57865091 podStartE2EDuration="7.097996669s" totalImagesPullingTime="5.519345759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:33:21.096573312 +0000 UTC m=+524.906120802" watchObservedRunningTime="2026-07-07 14:33:21.097996669 +0000 UTC m=+524.907544184" Jul 07 14:33:27.109784 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:27.109756 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-cvhmk" Jul 07 14:33:27.490695 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:27.490663 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:27.733950 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:27.733921 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5bmp2"] Jul 07 14:33:27.738720 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:27.738701 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-5bmp2" Jul 07 14:33:27.741476 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:27.741419 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4x47t\"" Jul 07 14:33:27.747735 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:27.747712 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5bmp2"] Jul 07 14:33:27.756746 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:27.756718 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn6n4\" (UniqueName: \"kubernetes.io/projected/8cfe4c55-3a30-44c5-b80a-abee6a620c12-kube-api-access-bn6n4\") pod \"authorino-5d4b67c446-5bmp2\" (UID: \"8cfe4c55-3a30-44c5-b80a-abee6a620c12\") " pod="kuadrant-system/authorino-5d4b67c446-5bmp2" Jul 07 14:33:27.858230 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:27.858192 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn6n4\" (UniqueName: \"kubernetes.io/projected/8cfe4c55-3a30-44c5-b80a-abee6a620c12-kube-api-access-bn6n4\") pod \"authorino-5d4b67c446-5bmp2\" (UID: \"8cfe4c55-3a30-44c5-b80a-abee6a620c12\") " pod="kuadrant-system/authorino-5d4b67c446-5bmp2" Jul 07 14:33:27.868208 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:27.868172 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn6n4\" (UniqueName: \"kubernetes.io/projected/8cfe4c55-3a30-44c5-b80a-abee6a620c12-kube-api-access-bn6n4\") pod \"authorino-5d4b67c446-5bmp2\" (UID: \"8cfe4c55-3a30-44c5-b80a-abee6a620c12\") " pod="kuadrant-system/authorino-5d4b67c446-5bmp2" Jul 07 14:33:27.950525 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:27.950490 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:27.998751 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:27.998666 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5bmp2"] Jul 07 14:33:27.998937 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:27.998926 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-5bmp2" Jul 07 14:33:28.022807 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.022778 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bc9cbfbcf-7shqz"] Jul 07 14:33:28.027283 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.027236 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bc9cbfbcf-7shqz" Jul 07 14:33:28.033066 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.033042 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bc9cbfbcf-7shqz"] Jul 07 14:33:28.060745 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.060719 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28n7c\" (UniqueName: \"kubernetes.io/projected/4e7baf03-6302-4bc1-a03a-946de9db68a0-kube-api-access-28n7c\") pod \"authorino-7bc9cbfbcf-7shqz\" (UID: \"4e7baf03-6302-4bc1-a03a-946de9db68a0\") " pod="kuadrant-system/authorino-7bc9cbfbcf-7shqz" Jul 07 14:33:28.134007 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.133981 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5bmp2"] Jul 07 14:33:28.136279 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:33:28.136250 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cfe4c55_3a30_44c5_b80a_abee6a620c12.slice/crio-f6fe759da5d61abb03d7f995371183c93e0d6e85595c894d6862f2f1d483ef96 WatchSource:0}: Error finding container f6fe759da5d61abb03d7f995371183c93e0d6e85595c894d6862f2f1d483ef96: Status 404 returned error can't find the container with id f6fe759da5d61abb03d7f995371183c93e0d6e85595c894d6862f2f1d483ef96 Jul 07 14:33:28.161628 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.161604 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28n7c\" (UniqueName: \"kubernetes.io/projected/4e7baf03-6302-4bc1-a03a-946de9db68a0-kube-api-access-28n7c\") pod \"authorino-7bc9cbfbcf-7shqz\" (UID: \"4e7baf03-6302-4bc1-a03a-946de9db68a0\") " pod="kuadrant-system/authorino-7bc9cbfbcf-7shqz" Jul 07 14:33:28.171707 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.171685 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28n7c\" (UniqueName: \"kubernetes.io/projected/4e7baf03-6302-4bc1-a03a-946de9db68a0-kube-api-access-28n7c\") pod \"authorino-7bc9cbfbcf-7shqz\" (UID: \"4e7baf03-6302-4bc1-a03a-946de9db68a0\") " pod="kuadrant-system/authorino-7bc9cbfbcf-7shqz" Jul 07 14:33:28.296891 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.296808 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bc9cbfbcf-7shqz"] Jul 07 14:33:28.297085 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.297072 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bc9cbfbcf-7shqz" Jul 07 14:33:28.325356 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.325329 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5c4d8c867-62hsh"] Jul 07 14:33:28.329808 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.329787 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c4d8c867-62hsh" Jul 07 14:33:28.333016 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.332995 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 07 14:33:28.337277 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.337256 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c4d8c867-62hsh"] Jul 07 14:33:28.363592 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.363565 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhh27\" (UniqueName: \"kubernetes.io/projected/a3234c84-f3bc-4ea6-a97a-676e1aec05a4-kube-api-access-hhh27\") pod \"authorino-5c4d8c867-62hsh\" (UID: \"a3234c84-f3bc-4ea6-a97a-676e1aec05a4\") " pod="kuadrant-system/authorino-5c4d8c867-62hsh" Jul 07 14:33:28.363725 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.363602 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a3234c84-f3bc-4ea6-a97a-676e1aec05a4-tls-cert\") pod \"authorino-5c4d8c867-62hsh\" (UID: \"a3234c84-f3bc-4ea6-a97a-676e1aec05a4\") " pod="kuadrant-system/authorino-5c4d8c867-62hsh" Jul 07 14:33:28.431216 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.431151 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bc9cbfbcf-7shqz"] Jul 07 14:33:28.433702 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:33:28.433676 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e7baf03_6302_4bc1_a03a_946de9db68a0.slice/crio-1fc98758f25660035a2c6f1c12fb46149be3a65da8bf06c2ea92358f741d5ae0 WatchSource:0}: Error finding container 1fc98758f25660035a2c6f1c12fb46149be3a65da8bf06c2ea92358f741d5ae0: Status 404 returned error can't find the container with id 1fc98758f25660035a2c6f1c12fb46149be3a65da8bf06c2ea92358f741d5ae0 Jul 07 14:33:28.464912 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.464875 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhh27\" (UniqueName: \"kubernetes.io/projected/a3234c84-f3bc-4ea6-a97a-676e1aec05a4-kube-api-access-hhh27\") pod \"authorino-5c4d8c867-62hsh\" (UID: \"a3234c84-f3bc-4ea6-a97a-676e1aec05a4\") " pod="kuadrant-system/authorino-5c4d8c867-62hsh" Jul 07 14:33:28.464912 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.464911 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a3234c84-f3bc-4ea6-a97a-676e1aec05a4-tls-cert\") pod \"authorino-5c4d8c867-62hsh\" (UID: \"a3234c84-f3bc-4ea6-a97a-676e1aec05a4\") " pod="kuadrant-system/authorino-5c4d8c867-62hsh" Jul 07 14:33:28.467601 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.467572 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a3234c84-f3bc-4ea6-a97a-676e1aec05a4-tls-cert\") pod \"authorino-5c4d8c867-62hsh\" (UID: \"a3234c84-f3bc-4ea6-a97a-676e1aec05a4\") " pod="kuadrant-system/authorino-5c4d8c867-62hsh" Jul 07 14:33:28.474500 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.474479 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhh27\" (UniqueName: \"kubernetes.io/projected/a3234c84-f3bc-4ea6-a97a-676e1aec05a4-kube-api-access-hhh27\") pod \"authorino-5c4d8c867-62hsh\" (UID: \"a3234c84-f3bc-4ea6-a97a-676e1aec05a4\") " pod="kuadrant-system/authorino-5c4d8c867-62hsh" Jul 07 14:33:28.641221 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.641132 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c4d8c867-62hsh" Jul 07 14:33:28.777934 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:28.777911 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5c4d8c867-62hsh"] Jul 07 14:33:28.780433 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:33:28.780366 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3234c84_f3bc_4ea6_a97a_676e1aec05a4.slice/crio-89a1f798d6eea164166231fc8806148b6f99d9ee00e8deb3c1d63e07f7f39be6 WatchSource:0}: Error finding container 89a1f798d6eea164166231fc8806148b6f99d9ee00e8deb3c1d63e07f7f39be6: Status 404 returned error can't find the container with id 89a1f798d6eea164166231fc8806148b6f99d9ee00e8deb3c1d63e07f7f39be6 Jul 07 14:33:29.115766 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:29.115730 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c4d8c867-62hsh" event={"ID":"a3234c84-f3bc-4ea6-a97a-676e1aec05a4","Type":"ContainerStarted","Data":"89a1f798d6eea164166231fc8806148b6f99d9ee00e8deb3c1d63e07f7f39be6"} Jul 07 14:33:29.117240 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:29.117210 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bc9cbfbcf-7shqz" event={"ID":"4e7baf03-6302-4bc1-a03a-946de9db68a0","Type":"ContainerStarted","Data":"3b19cdf16b8c7a3a9aaaf4049001347b29d44adf82b397c708c33b23b3abda47"} Jul 07 14:33:29.117399 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:29.117249 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bc9cbfbcf-7shqz" event={"ID":"4e7baf03-6302-4bc1-a03a-946de9db68a0","Type":"ContainerStarted","Data":"1fc98758f25660035a2c6f1c12fb46149be3a65da8bf06c2ea92358f741d5ae0"} Jul 07 14:33:29.117399 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:29.117267 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-7bc9cbfbcf-7shqz" podUID="4e7baf03-6302-4bc1-a03a-946de9db68a0" containerName="authorino" containerID="cri-o://3b19cdf16b8c7a3a9aaaf4049001347b29d44adf82b397c708c33b23b3abda47" gracePeriod=30 Jul 07 14:33:29.118921 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:29.118895 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-5bmp2" event={"ID":"8cfe4c55-3a30-44c5-b80a-abee6a620c12","Type":"ContainerStarted","Data":"fe4e50d35ce86b6f2314e30ebe57e38b1deca925ed36228c4a55f78584f5614f"} Jul 07 14:33:29.119031 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:29.118938 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-5bmp2" event={"ID":"8cfe4c55-3a30-44c5-b80a-abee6a620c12","Type":"ContainerStarted","Data":"f6fe759da5d61abb03d7f995371183c93e0d6e85595c894d6862f2f1d483ef96"} Jul 07 14:33:29.119031 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:29.118902 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-5bmp2" podUID="8cfe4c55-3a30-44c5-b80a-abee6a620c12" containerName="authorino" containerID="cri-o://fe4e50d35ce86b6f2314e30ebe57e38b1deca925ed36228c4a55f78584f5614f" gracePeriod=30 Jul 07 14:33:29.135853 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:29.135810 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bc9cbfbcf-7shqz" podStartSLOduration=0.724378219 podStartE2EDuration="1.135795482s" totalImagesPullingTime="411.417263ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:33:29.134565788 +0000 UTC m=+532.944113279" watchObservedRunningTime="2026-07-07 14:33:29.135795482 +0000 UTC m=+532.945342975" Jul 07 14:33:29.151763 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:29.151720 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-5bmp2" podStartSLOduration=1.801989706 podStartE2EDuration="2.151705081s" totalImagesPullingTime="349.715375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:33:29.150133734 +0000 UTC m=+532.959681224" watchObservedRunningTime="2026-07-07 14:33:29.151705081 +0000 UTC m=+532.961252571" Jul 07 14:33:29.386437 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:29.386406 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-5bmp2" Jul 07 14:33:29.389271 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:29.389246 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bc9cbfbcf-7shqz" Jul 07 14:33:29.473848 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:29.473763 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28n7c\" (UniqueName: \"kubernetes.io/projected/4e7baf03-6302-4bc1-a03a-946de9db68a0-kube-api-access-28n7c\") pod \"4e7baf03-6302-4bc1-a03a-946de9db68a0\" (UID: \"4e7baf03-6302-4bc1-a03a-946de9db68a0\") " Jul 07 14:33:29.473848 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:29.473809 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bn6n4\" (UniqueName: \"kubernetes.io/projected/8cfe4c55-3a30-44c5-b80a-abee6a620c12-kube-api-access-bn6n4\") pod \"8cfe4c55-3a30-44c5-b80a-abee6a620c12\" (UID: \"8cfe4c55-3a30-44c5-b80a-abee6a620c12\") " Jul 07 14:33:29.475826 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:29.475797 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfe4c55-3a30-44c5-b80a-abee6a620c12-kube-api-access-bn6n4" (OuterVolumeSpecName: "kube-api-access-bn6n4") pod "8cfe4c55-3a30-44c5-b80a-abee6a620c12" (UID: "8cfe4c55-3a30-44c5-b80a-abee6a620c12"). InnerVolumeSpecName "kube-api-access-bn6n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:33:29.475936 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:29.475883 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7baf03-6302-4bc1-a03a-946de9db68a0-kube-api-access-28n7c" (OuterVolumeSpecName: "kube-api-access-28n7c") pod "4e7baf03-6302-4bc1-a03a-946de9db68a0" (UID: "4e7baf03-6302-4bc1-a03a-946de9db68a0"). InnerVolumeSpecName "kube-api-access-28n7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:33:29.575289 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:29.575250 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28n7c\" (UniqueName: \"kubernetes.io/projected/4e7baf03-6302-4bc1-a03a-946de9db68a0-kube-api-access-28n7c\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:33:29.575289 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:29.575282 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn6n4\" (UniqueName: \"kubernetes.io/projected/8cfe4c55-3a30-44c5-b80a-abee6a620c12-kube-api-access-bn6n4\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:33:30.125750 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.125710 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c4d8c867-62hsh" event={"ID":"a3234c84-f3bc-4ea6-a97a-676e1aec05a4","Type":"ContainerStarted","Data":"8531e0563b099f917f99b3ebd131aff966ed3fa3ac2bd0a5ce9b5ab7e25183ce"} Jul 07 14:33:30.126920 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.126894 2562 generic.go:356] "Generic (PLEG): container finished" podID="4e7baf03-6302-4bc1-a03a-946de9db68a0" containerID="3b19cdf16b8c7a3a9aaaf4049001347b29d44adf82b397c708c33b23b3abda47" exitCode=0 Jul 07 14:33:30.127047 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.126946 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bc9cbfbcf-7shqz" Jul 07 14:33:30.127047 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.126961 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bc9cbfbcf-7shqz" event={"ID":"4e7baf03-6302-4bc1-a03a-946de9db68a0","Type":"ContainerDied","Data":"3b19cdf16b8c7a3a9aaaf4049001347b29d44adf82b397c708c33b23b3abda47"} Jul 07 14:33:30.127047 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.126995 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bc9cbfbcf-7shqz" event={"ID":"4e7baf03-6302-4bc1-a03a-946de9db68a0","Type":"ContainerDied","Data":"1fc98758f25660035a2c6f1c12fb46149be3a65da8bf06c2ea92358f741d5ae0"} Jul 07 14:33:30.127047 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.127017 2562 scope.go:117] "RemoveContainer" containerID="3b19cdf16b8c7a3a9aaaf4049001347b29d44adf82b397c708c33b23b3abda47" Jul 07 14:33:30.128266 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.128236 2562 generic.go:356] "Generic (PLEG): container finished" podID="8cfe4c55-3a30-44c5-b80a-abee6a620c12" containerID="fe4e50d35ce86b6f2314e30ebe57e38b1deca925ed36228c4a55f78584f5614f" exitCode=0 Jul 07 14:33:30.128395 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.128278 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-5bmp2" event={"ID":"8cfe4c55-3a30-44c5-b80a-abee6a620c12","Type":"ContainerDied","Data":"fe4e50d35ce86b6f2314e30ebe57e38b1deca925ed36228c4a55f78584f5614f"} Jul 07 14:33:30.128395 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.128309 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-5bmp2" event={"ID":"8cfe4c55-3a30-44c5-b80a-abee6a620c12","Type":"ContainerDied","Data":"f6fe759da5d61abb03d7f995371183c93e0d6e85595c894d6862f2f1d483ef96"} Jul 07 14:33:30.128395 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.128284 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-5bmp2" Jul 07 14:33:30.135504 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.135482 2562 scope.go:117] "RemoveContainer" containerID="3b19cdf16b8c7a3a9aaaf4049001347b29d44adf82b397c708c33b23b3abda47" Jul 07 14:33:30.135791 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:33:30.135768 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b19cdf16b8c7a3a9aaaf4049001347b29d44adf82b397c708c33b23b3abda47\": container with ID starting with 3b19cdf16b8c7a3a9aaaf4049001347b29d44adf82b397c708c33b23b3abda47 not found: ID does not exist" containerID="3b19cdf16b8c7a3a9aaaf4049001347b29d44adf82b397c708c33b23b3abda47" Jul 07 14:33:30.135899 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.135801 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b19cdf16b8c7a3a9aaaf4049001347b29d44adf82b397c708c33b23b3abda47"} err="failed to get container status \"3b19cdf16b8c7a3a9aaaf4049001347b29d44adf82b397c708c33b23b3abda47\": rpc error: code = NotFound desc = could not find container \"3b19cdf16b8c7a3a9aaaf4049001347b29d44adf82b397c708c33b23b3abda47\": container with ID starting with 3b19cdf16b8c7a3a9aaaf4049001347b29d44adf82b397c708c33b23b3abda47 not found: ID does not exist" Jul 07 14:33:30.135899 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.135825 2562 scope.go:117] "RemoveContainer" containerID="fe4e50d35ce86b6f2314e30ebe57e38b1deca925ed36228c4a55f78584f5614f" Jul 07 14:33:30.143822 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.143801 2562 scope.go:117] "RemoveContainer" containerID="fe4e50d35ce86b6f2314e30ebe57e38b1deca925ed36228c4a55f78584f5614f" Jul 07 14:33:30.144451 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:33:30.144268 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe4e50d35ce86b6f2314e30ebe57e38b1deca925ed36228c4a55f78584f5614f\": container with ID starting with fe4e50d35ce86b6f2314e30ebe57e38b1deca925ed36228c4a55f78584f5614f not found: ID does not exist" containerID="fe4e50d35ce86b6f2314e30ebe57e38b1deca925ed36228c4a55f78584f5614f" Jul 07 14:33:30.144451 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.144304 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe4e50d35ce86b6f2314e30ebe57e38b1deca925ed36228c4a55f78584f5614f"} err="failed to get container status \"fe4e50d35ce86b6f2314e30ebe57e38b1deca925ed36228c4a55f78584f5614f\": rpc error: code = NotFound desc = could not find container \"fe4e50d35ce86b6f2314e30ebe57e38b1deca925ed36228c4a55f78584f5614f\": container with ID starting with fe4e50d35ce86b6f2314e30ebe57e38b1deca925ed36228c4a55f78584f5614f not found: ID does not exist" Jul 07 14:33:30.145341 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.145304 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5c4d8c867-62hsh" podStartSLOduration=1.761352691 podStartE2EDuration="2.145290175s" totalImagesPullingTime="383.937484ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:33:30.143721762 +0000 UTC m=+533.953269265" watchObservedRunningTime="2026-07-07 14:33:30.145290175 +0000 UTC m=+533.954837665" Jul 07 14:33:30.161193 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.161158 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bc9cbfbcf-7shqz"] Jul 07 14:33:30.164719 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.164695 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bc9cbfbcf-7shqz"] Jul 07 14:33:30.179106 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.179078 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5bmp2"] Jul 07 14:33:30.187987 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.187956 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-5bmp2"] Jul 07 14:33:30.706220 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.706185 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e7baf03-6302-4bc1-a03a-946de9db68a0" path="/var/lib/kubelet/pods/4e7baf03-6302-4bc1-a03a-946de9db68a0/volumes" Jul 07 14:33:30.706563 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:30.706550 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cfe4c55-3a30-44c5-b80a-abee6a620c12" path="/var/lib/kubelet/pods/8cfe4c55-3a30-44c5-b80a-abee6a620c12/volumes" Jul 07 14:33:34.981060 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:34.981026 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:35.082682 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.082649 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:35.180956 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.180921 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:35.189797 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.189766 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6495c9587b-n2vbf"] Jul 07 14:33:35.190135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.190113 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e7baf03-6302-4bc1-a03a-946de9db68a0" containerName="authorino" Jul 07 14:33:35.190135 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.190128 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e7baf03-6302-4bc1-a03a-946de9db68a0" containerName="authorino" Jul 07 14:33:35.190265 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.190153 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cfe4c55-3a30-44c5-b80a-abee6a620c12" containerName="authorino" Jul 07 14:33:35.190265 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.190159 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cfe4c55-3a30-44c5-b80a-abee6a620c12" containerName="authorino" Jul 07 14:33:35.190265 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.190223 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e7baf03-6302-4bc1-a03a-946de9db68a0" containerName="authorino" Jul 07 14:33:35.190265 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.190233 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cfe4c55-3a30-44c5-b80a-abee6a620c12" containerName="authorino" Jul 07 14:33:35.364306 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.364215 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6495c9587b-n2vbf"] Jul 07 14:33:35.364480 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.364335 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6495c9587b-n2vbf" Jul 07 14:33:35.368085 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.368064 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 07 14:33:35.368085 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.368085 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 07 14:33:35.368264 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.368085 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-44wmw\"" Jul 07 14:33:35.426556 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.426514 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnxc9\" (UniqueName: \"kubernetes.io/projected/01c9d772-39f6-4c25-bdeb-944ccf78627b-kube-api-access-wnxc9\") pod \"maas-controller-6495c9587b-n2vbf\" (UID: \"01c9d772-39f6-4c25-bdeb-944ccf78627b\") " pod="opendatahub/maas-controller-6495c9587b-n2vbf" Jul 07 14:33:35.426731 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.426654 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01c9d772-39f6-4c25-bdeb-944ccf78627b-webhook-cert\") pod \"maas-controller-6495c9587b-n2vbf\" (UID: \"01c9d772-39f6-4c25-bdeb-944ccf78627b\") " pod="opendatahub/maas-controller-6495c9587b-n2vbf" Jul 07 14:33:35.527512 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.527473 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01c9d772-39f6-4c25-bdeb-944ccf78627b-webhook-cert\") pod \"maas-controller-6495c9587b-n2vbf\" (UID: \"01c9d772-39f6-4c25-bdeb-944ccf78627b\") " pod="opendatahub/maas-controller-6495c9587b-n2vbf" Jul 07 14:33:35.527676 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.527525 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnxc9\" (UniqueName: \"kubernetes.io/projected/01c9d772-39f6-4c25-bdeb-944ccf78627b-kube-api-access-wnxc9\") pod \"maas-controller-6495c9587b-n2vbf\" (UID: \"01c9d772-39f6-4c25-bdeb-944ccf78627b\") " pod="opendatahub/maas-controller-6495c9587b-n2vbf" Jul 07 14:33:35.529939 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.529914 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01c9d772-39f6-4c25-bdeb-944ccf78627b-webhook-cert\") pod \"maas-controller-6495c9587b-n2vbf\" (UID: \"01c9d772-39f6-4c25-bdeb-944ccf78627b\") " pod="opendatahub/maas-controller-6495c9587b-n2vbf" Jul 07 14:33:35.539186 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.539162 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnxc9\" (UniqueName: \"kubernetes.io/projected/01c9d772-39f6-4c25-bdeb-944ccf78627b-kube-api-access-wnxc9\") pod \"maas-controller-6495c9587b-n2vbf\" (UID: \"01c9d772-39f6-4c25-bdeb-944ccf78627b\") " pod="opendatahub/maas-controller-6495c9587b-n2vbf" Jul 07 14:33:35.674946 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.674906 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6495c9587b-n2vbf" Jul 07 14:33:35.855463 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:35.855434 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6495c9587b-n2vbf"] Jul 07 14:33:35.857484 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:33:35.857455 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c9d772_39f6_4c25_bdeb_944ccf78627b.slice/crio-2360dd22940a41d50e24e76c16eea870f1c0afc92c3b4fbe482471cf6cf1c249 WatchSource:0}: Error finding container 2360dd22940a41d50e24e76c16eea870f1c0afc92c3b4fbe482471cf6cf1c249: Status 404 returned error can't find the container with id 2360dd22940a41d50e24e76c16eea870f1c0afc92c3b4fbe482471cf6cf1c249 Jul 07 14:33:36.160783 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:36.160746 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6495c9587b-n2vbf" event={"ID":"01c9d772-39f6-4c25-bdeb-944ccf78627b","Type":"ContainerStarted","Data":"2360dd22940a41d50e24e76c16eea870f1c0afc92c3b4fbe482471cf6cf1c249"} Jul 07 14:33:37.661181 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:37.661159 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 07 14:33:37.847525 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:37.847481 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:37.939567 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:37.939450 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:33:38.174533 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:38.174500 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6495c9587b-n2vbf" event={"ID":"01c9d772-39f6-4c25-bdeb-944ccf78627b","Type":"ContainerStarted","Data":"8f0569a04392d669e215b87d7d29b0df40b51a239089cdd9c79cf2d28b30fc95"} Jul 07 14:33:38.174731 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:38.174597 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6495c9587b-n2vbf" Jul 07 14:33:38.211999 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:38.211880 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6495c9587b-n2vbf" podStartSLOduration=1.412170852 podStartE2EDuration="3.211860932s" totalImagesPullingTime="1.79969008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:33:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:33:38.210118006 +0000 UTC m=+542.019665534" watchObservedRunningTime="2026-07-07 14:33:38.211860932 +0000 UTC m=+542.021408458" Jul 07 14:33:49.189060 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:33:49.189027 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6495c9587b-n2vbf" Jul 07 14:34:07.849757 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:07.849724 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:08.340096 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:08.340054 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:08.500552 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:08.500516 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-64f64495c-wq6t7"] Jul 07 14:34:08.505286 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:08.505271 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-64f64495c-wq6t7" Jul 07 14:34:08.509303 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:08.509283 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-cx6pw\"" Jul 07 14:34:08.509629 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:08.509615 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 07 14:34:08.528620 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:08.528596 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a127701-b325-405f-a500-348b93eefae7-maas-api-tls\") pod \"maas-api-64f64495c-wq6t7\" (UID: \"3a127701-b325-405f-a500-348b93eefae7\") " pod="opendatahub/maas-api-64f64495c-wq6t7" Jul 07 14:34:08.528778 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:08.528692 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tldl\" (UniqueName: \"kubernetes.io/projected/3a127701-b325-405f-a500-348b93eefae7-kube-api-access-6tldl\") pod \"maas-api-64f64495c-wq6t7\" (UID: \"3a127701-b325-405f-a500-348b93eefae7\") " pod="opendatahub/maas-api-64f64495c-wq6t7" Jul 07 14:34:08.528853 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:08.528773 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a127701-b325-405f-a500-348b93eefae7-openshift-service-ca\") pod \"maas-api-64f64495c-wq6t7\" (UID: \"3a127701-b325-405f-a500-348b93eefae7\") " pod="opendatahub/maas-api-64f64495c-wq6t7" Jul 07 14:34:08.531020 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:08.530993 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-64f64495c-wq6t7"] Jul 07 14:34:08.629619 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:08.629518 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a127701-b325-405f-a500-348b93eefae7-maas-api-tls\") pod \"maas-api-64f64495c-wq6t7\" (UID: \"3a127701-b325-405f-a500-348b93eefae7\") " pod="opendatahub/maas-api-64f64495c-wq6t7" Jul 07 14:34:08.629619 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:08.629588 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tldl\" (UniqueName: \"kubernetes.io/projected/3a127701-b325-405f-a500-348b93eefae7-kube-api-access-6tldl\") pod \"maas-api-64f64495c-wq6t7\" (UID: \"3a127701-b325-405f-a500-348b93eefae7\") " pod="opendatahub/maas-api-64f64495c-wq6t7" Jul 07 14:34:08.629810 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:08.629635 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a127701-b325-405f-a500-348b93eefae7-openshift-service-ca\") pod \"maas-api-64f64495c-wq6t7\" (UID: \"3a127701-b325-405f-a500-348b93eefae7\") " pod="opendatahub/maas-api-64f64495c-wq6t7" Jul 07 14:34:08.630294 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:08.630267 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a127701-b325-405f-a500-348b93eefae7-openshift-service-ca\") pod \"maas-api-64f64495c-wq6t7\" (UID: \"3a127701-b325-405f-a500-348b93eefae7\") " pod="opendatahub/maas-api-64f64495c-wq6t7" Jul 07 14:34:08.631986 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:08.631968 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a127701-b325-405f-a500-348b93eefae7-maas-api-tls\") pod \"maas-api-64f64495c-wq6t7\" (UID: \"3a127701-b325-405f-a500-348b93eefae7\") " pod="opendatahub/maas-api-64f64495c-wq6t7" Jul 07 14:34:08.641650 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:08.641627 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tldl\" (UniqueName: \"kubernetes.io/projected/3a127701-b325-405f-a500-348b93eefae7-kube-api-access-6tldl\") pod \"maas-api-64f64495c-wq6t7\" (UID: \"3a127701-b325-405f-a500-348b93eefae7\") " pod="opendatahub/maas-api-64f64495c-wq6t7" Jul 07 14:34:08.654629 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:08.654601 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:08.815704 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:08.815671 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-64f64495c-wq6t7" Jul 07 14:34:08.960604 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:08.960565 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-64f64495c-wq6t7"] Jul 07 14:34:08.963478 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:34:08.963442 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a127701_b325_405f_a500_348b93eefae7.slice/crio-8c96977ea4609b7d845c35edbc100937689e9305a7940167e1f4ef9631b34899 WatchSource:0}: Error finding container 8c96977ea4609b7d845c35edbc100937689e9305a7940167e1f4ef9631b34899: Status 404 returned error can't find the container with id 8c96977ea4609b7d845c35edbc100937689e9305a7940167e1f4ef9631b34899 Jul 07 14:34:08.999070 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:08.999037 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:09.097986 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:09.097935 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:09.206842 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:09.206767 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-5vgm2"] Jul 07 14:34:09.211261 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:09.211239 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-5vgm2" Jul 07 14:34:09.213851 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:09.213828 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lffvg\"" Jul 07 14:34:09.214028 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:09.214011 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 07 14:34:09.226870 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:09.226843 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-5vgm2"] Jul 07 14:34:09.234471 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:09.234445 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f202d8f1-a537-4765-9352-8a229fb5e65f-plugins-config-volume\") pod \"payload-pre-processing-7964457696-5vgm2\" (UID: \"f202d8f1-a537-4765-9352-8a229fb5e65f\") " pod="openshift-ingress/payload-pre-processing-7964457696-5vgm2" Jul 07 14:34:09.234607 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:09.234518 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmlkh\" (UniqueName: \"kubernetes.io/projected/f202d8f1-a537-4765-9352-8a229fb5e65f-kube-api-access-rmlkh\") pod \"payload-pre-processing-7964457696-5vgm2\" (UID: \"f202d8f1-a537-4765-9352-8a229fb5e65f\") " pod="openshift-ingress/payload-pre-processing-7964457696-5vgm2" Jul 07 14:34:09.321526 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:09.321489 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-64f64495c-wq6t7" event={"ID":"3a127701-b325-405f-a500-348b93eefae7","Type":"ContainerStarted","Data":"8c96977ea4609b7d845c35edbc100937689e9305a7940167e1f4ef9631b34899"} Jul 07 14:34:09.335141 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:09.335099 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmlkh\" (UniqueName: \"kubernetes.io/projected/f202d8f1-a537-4765-9352-8a229fb5e65f-kube-api-access-rmlkh\") pod \"payload-pre-processing-7964457696-5vgm2\" (UID: \"f202d8f1-a537-4765-9352-8a229fb5e65f\") " pod="openshift-ingress/payload-pre-processing-7964457696-5vgm2" Jul 07 14:34:09.335315 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:09.335174 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f202d8f1-a537-4765-9352-8a229fb5e65f-plugins-config-volume\") pod \"payload-pre-processing-7964457696-5vgm2\" (UID: \"f202d8f1-a537-4765-9352-8a229fb5e65f\") " pod="openshift-ingress/payload-pre-processing-7964457696-5vgm2" Jul 07 14:34:09.335888 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:09.335869 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f202d8f1-a537-4765-9352-8a229fb5e65f-plugins-config-volume\") pod \"payload-pre-processing-7964457696-5vgm2\" (UID: \"f202d8f1-a537-4765-9352-8a229fb5e65f\") " pod="openshift-ingress/payload-pre-processing-7964457696-5vgm2" Jul 07 14:34:09.346218 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:09.346185 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmlkh\" (UniqueName: \"kubernetes.io/projected/f202d8f1-a537-4765-9352-8a229fb5e65f-kube-api-access-rmlkh\") pod \"payload-pre-processing-7964457696-5vgm2\" (UID: \"f202d8f1-a537-4765-9352-8a229fb5e65f\") " pod="openshift-ingress/payload-pre-processing-7964457696-5vgm2" Jul 07 14:34:09.521758 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:09.521667 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-5vgm2" Jul 07 14:34:09.663244 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:09.663128 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-5vgm2"] Jul 07 14:34:09.665403 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:34:09.665361 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf202d8f1_a537_4765_9352_8a229fb5e65f.slice/crio-d02fc9fe4baee8baee68f41d7c4201d6214a8fd104b6da0e282f0a8f5520c858 WatchSource:0}: Error finding container d02fc9fe4baee8baee68f41d7c4201d6214a8fd104b6da0e282f0a8f5520c858: Status 404 returned error can't find the container with id d02fc9fe4baee8baee68f41d7c4201d6214a8fd104b6da0e282f0a8f5520c858 Jul 07 14:34:10.327032 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:10.327002 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-5vgm2" event={"ID":"f202d8f1-a537-4765-9352-8a229fb5e65f","Type":"ContainerStarted","Data":"d02fc9fe4baee8baee68f41d7c4201d6214a8fd104b6da0e282f0a8f5520c858"} Jul 07 14:34:10.886014 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:10.885967 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:11.333584 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:11.333537 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-64f64495c-wq6t7" event={"ID":"3a127701-b325-405f-a500-348b93eefae7","Type":"ContainerStarted","Data":"22a1d014e44ebb971de92cbf5729c738e4a374347f03d625aecfb6656a4f7b16"} Jul 07 14:34:11.333970 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:11.333604 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-64f64495c-wq6t7" Jul 07 14:34:11.352630 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:11.352578 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-64f64495c-wq6t7" podStartSLOduration=1.466530018 podStartE2EDuration="3.352558944s" totalImagesPullingTime="1.886028926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:34:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:11.351845894 +0000 UTC m=+575.161393384" watchObservedRunningTime="2026-07-07 14:34:11.352558944 +0000 UTC m=+575.162106434" Jul 07 14:34:13.344526 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:13.344492 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-5vgm2" event={"ID":"f202d8f1-a537-4765-9352-8a229fb5e65f","Type":"ContainerStarted","Data":"8015629285d13b798584e90d9ba1aa359e1b0d93ca018a3e52cd5a7ee768059a"} Jul 07 14:34:13.344920 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:13.344571 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-5vgm2" Jul 07 14:34:13.364364 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:13.364307 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-5vgm2" podStartSLOduration=1.348467242 podStartE2EDuration="4.364289768s" totalImagesPullingTime="3.015822526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:34:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:13.362268546 +0000 UTC m=+577.171816036" watchObservedRunningTime="2026-07-07 14:34:13.364289768 +0000 UTC m=+577.173837293" Jul 07 14:34:17.343734 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:17.343703 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-64f64495c-wq6t7" Jul 07 14:34:18.513284 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:18.513250 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:18.607690 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:18.607657 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:21.230698 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:21.230667 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:24.352084 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:24.352054 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-5vgm2" Jul 07 14:34:25.143123 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:25.143086 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:25.239065 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:25.239032 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:31.880718 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:31.880680 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:44.348305 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.348264 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9"] Jul 07 14:34:44.352206 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.352183 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.354872 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.354849 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 07 14:34:44.354993 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.354854 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 07 14:34:44.354993 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.354918 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ms4lq\"" Jul 07 14:34:44.355794 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.355775 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 07 14:34:44.360815 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.360788 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64z2s\" (UniqueName: \"kubernetes.io/projected/075d5305-a170-4da4-a00d-2fbdbb5d5a6c-kube-api-access-64z2s\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6p7w9\" (UID: \"075d5305-a170-4da4-a00d-2fbdbb5d5a6c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.360927 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.360846 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/075d5305-a170-4da4-a00d-2fbdbb5d5a6c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6p7w9\" (UID: \"075d5305-a170-4da4-a00d-2fbdbb5d5a6c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.360927 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.360869 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/075d5305-a170-4da4-a00d-2fbdbb5d5a6c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6p7w9\" (UID: \"075d5305-a170-4da4-a00d-2fbdbb5d5a6c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.360927 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.360904 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/075d5305-a170-4da4-a00d-2fbdbb5d5a6c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6p7w9\" (UID: \"075d5305-a170-4da4-a00d-2fbdbb5d5a6c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.361088 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.360950 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/075d5305-a170-4da4-a00d-2fbdbb5d5a6c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6p7w9\" (UID: \"075d5305-a170-4da4-a00d-2fbdbb5d5a6c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.361088 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.361022 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/075d5305-a170-4da4-a00d-2fbdbb5d5a6c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6p7w9\" (UID: \"075d5305-a170-4da4-a00d-2fbdbb5d5a6c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.367388 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.367308 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9"] Jul 07 14:34:44.380681 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.380650 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:44.461692 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.461656 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/075d5305-a170-4da4-a00d-2fbdbb5d5a6c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6p7w9\" (UID: \"075d5305-a170-4da4-a00d-2fbdbb5d5a6c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.461692 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.461693 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64z2s\" (UniqueName: \"kubernetes.io/projected/075d5305-a170-4da4-a00d-2fbdbb5d5a6c-kube-api-access-64z2s\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6p7w9\" (UID: \"075d5305-a170-4da4-a00d-2fbdbb5d5a6c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.461963 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.461731 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/075d5305-a170-4da4-a00d-2fbdbb5d5a6c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6p7w9\" (UID: \"075d5305-a170-4da4-a00d-2fbdbb5d5a6c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.461963 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.461748 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/075d5305-a170-4da4-a00d-2fbdbb5d5a6c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6p7w9\" (UID: \"075d5305-a170-4da4-a00d-2fbdbb5d5a6c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.461963 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.461778 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/075d5305-a170-4da4-a00d-2fbdbb5d5a6c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6p7w9\" (UID: \"075d5305-a170-4da4-a00d-2fbdbb5d5a6c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.461963 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.461819 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/075d5305-a170-4da4-a00d-2fbdbb5d5a6c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6p7w9\" (UID: \"075d5305-a170-4da4-a00d-2fbdbb5d5a6c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.462208 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.462185 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/075d5305-a170-4da4-a00d-2fbdbb5d5a6c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6p7w9\" (UID: \"075d5305-a170-4da4-a00d-2fbdbb5d5a6c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.462262 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.462241 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/075d5305-a170-4da4-a00d-2fbdbb5d5a6c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6p7w9\" (UID: \"075d5305-a170-4da4-a00d-2fbdbb5d5a6c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.462303 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.462249 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/075d5305-a170-4da4-a00d-2fbdbb5d5a6c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6p7w9\" (UID: \"075d5305-a170-4da4-a00d-2fbdbb5d5a6c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.464144 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.464120 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/075d5305-a170-4da4-a00d-2fbdbb5d5a6c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6p7w9\" (UID: \"075d5305-a170-4da4-a00d-2fbdbb5d5a6c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.464558 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.464539 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/075d5305-a170-4da4-a00d-2fbdbb5d5a6c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6p7w9\" (UID: \"075d5305-a170-4da4-a00d-2fbdbb5d5a6c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.470726 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.470704 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64z2s\" (UniqueName: \"kubernetes.io/projected/075d5305-a170-4da4-a00d-2fbdbb5d5a6c-kube-api-access-64z2s\") pod \"e2e-distinct-simulated-kserve-59975d69f9-6p7w9\" (UID: \"075d5305-a170-4da4-a00d-2fbdbb5d5a6c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.663153 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.663113 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:44.803567 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:44.803544 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9"] Jul 07 14:34:44.806022 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:34:44.805994 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod075d5305_a170_4da4_a00d_2fbdbb5d5a6c.slice/crio-906ae1d7e75296dc0b1d08eab18b26c8e4c41353ea52372b20b9711f2cef2977 WatchSource:0}: Error finding container 906ae1d7e75296dc0b1d08eab18b26c8e4c41353ea52372b20b9711f2cef2977: Status 404 returned error can't find the container with id 906ae1d7e75296dc0b1d08eab18b26c8e4c41353ea52372b20b9711f2cef2977 Jul 07 14:34:45.501225 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:45.501143 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" event={"ID":"075d5305-a170-4da4-a00d-2fbdbb5d5a6c","Type":"ContainerStarted","Data":"906ae1d7e75296dc0b1d08eab18b26c8e4c41353ea52372b20b9711f2cef2977"} Jul 07 14:34:46.507486 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:46.507451 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" event={"ID":"075d5305-a170-4da4-a00d-2fbdbb5d5a6c","Type":"ContainerStarted","Data":"0cf57386a4e7f6085364bda11eda3b4107209e9c4933b2c943f51799470ba6d0"} Jul 07 14:34:46.528020 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:46.527969 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" podStartSLOduration=1.261715632 podStartE2EDuration="2.527951314s" totalImagesPullingTime="1.266235682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:34:46.526610477 +0000 UTC m=+610.336157993" watchObservedRunningTime="2026-07-07 14:34:46.527951314 +0000 UTC m=+610.337498810" Jul 07 14:34:49.896702 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:49.896669 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:34:54.663460 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:54.663418 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:54.663931 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:54.663694 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:54.676104 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:54.676080 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:34:55.565039 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:34:55.565011 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-6p7w9" Jul 07 14:35:09.969652 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:09.969621 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w"] Jul 07 14:35:09.973147 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:09.973125 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:09.975770 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:09.975751 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 07 14:35:09.987100 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:09.987075 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w"] Jul 07 14:35:10.003161 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.003132 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:10.096764 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.096733 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47d7d674-b461-4be4-ab59-1beff66acbda-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w\" (UID: \"47d7d674-b461-4be4-ab59-1beff66acbda\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:10.096764 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.096775 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47d7d674-b461-4be4-ab59-1beff66acbda-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w\" (UID: \"47d7d674-b461-4be4-ab59-1beff66acbda\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:10.096991 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.096801 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47d7d674-b461-4be4-ab59-1beff66acbda-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w\" (UID: \"47d7d674-b461-4be4-ab59-1beff66acbda\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:10.096991 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.096919 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qdxk\" (UniqueName: \"kubernetes.io/projected/47d7d674-b461-4be4-ab59-1beff66acbda-kube-api-access-6qdxk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w\" (UID: \"47d7d674-b461-4be4-ab59-1beff66acbda\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:10.096991 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.096967 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47d7d674-b461-4be4-ab59-1beff66acbda-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w\" (UID: \"47d7d674-b461-4be4-ab59-1beff66acbda\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:10.096991 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.096989 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47d7d674-b461-4be4-ab59-1beff66acbda-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w\" (UID: \"47d7d674-b461-4be4-ab59-1beff66acbda\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:10.198432 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.198363 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47d7d674-b461-4be4-ab59-1beff66acbda-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w\" (UID: \"47d7d674-b461-4be4-ab59-1beff66acbda\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:10.198432 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.198436 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47d7d674-b461-4be4-ab59-1beff66acbda-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w\" (UID: \"47d7d674-b461-4be4-ab59-1beff66acbda\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:10.198682 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.198459 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47d7d674-b461-4be4-ab59-1beff66acbda-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w\" (UID: \"47d7d674-b461-4be4-ab59-1beff66acbda\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:10.198682 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.198502 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qdxk\" (UniqueName: \"kubernetes.io/projected/47d7d674-b461-4be4-ab59-1beff66acbda-kube-api-access-6qdxk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w\" (UID: \"47d7d674-b461-4be4-ab59-1beff66acbda\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:10.198682 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.198545 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47d7d674-b461-4be4-ab59-1beff66acbda-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w\" (UID: \"47d7d674-b461-4be4-ab59-1beff66acbda\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:10.198682 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.198575 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47d7d674-b461-4be4-ab59-1beff66acbda-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w\" (UID: \"47d7d674-b461-4be4-ab59-1beff66acbda\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:10.198877 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.198795 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47d7d674-b461-4be4-ab59-1beff66acbda-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w\" (UID: \"47d7d674-b461-4be4-ab59-1beff66acbda\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:10.198988 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.198968 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47d7d674-b461-4be4-ab59-1beff66acbda-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w\" (UID: \"47d7d674-b461-4be4-ab59-1beff66acbda\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:10.199088 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.199060 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47d7d674-b461-4be4-ab59-1beff66acbda-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w\" (UID: \"47d7d674-b461-4be4-ab59-1beff66acbda\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:10.200818 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.200796 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47d7d674-b461-4be4-ab59-1beff66acbda-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w\" (UID: \"47d7d674-b461-4be4-ab59-1beff66acbda\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:10.201043 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.201026 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47d7d674-b461-4be4-ab59-1beff66acbda-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w\" (UID: \"47d7d674-b461-4be4-ab59-1beff66acbda\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:10.207462 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.207437 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qdxk\" (UniqueName: \"kubernetes.io/projected/47d7d674-b461-4be4-ab59-1beff66acbda-kube-api-access-6qdxk\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w\" (UID: \"47d7d674-b461-4be4-ab59-1beff66acbda\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:10.283103 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.283016 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:10.423712 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.423670 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w"] Jul 07 14:35:10.426913 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:35:10.426881 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d7d674_b461_4be4_ab59_1beff66acbda.slice/crio-4e6c90919f31d85fa90be8455011ed907f5579d9383f8448a38c4484ae804dfe WatchSource:0}: Error finding container 4e6c90919f31d85fa90be8455011ed907f5579d9383f8448a38c4484ae804dfe: Status 404 returned error can't find the container with id 4e6c90919f31d85fa90be8455011ed907f5579d9383f8448a38c4484ae804dfe Jul 07 14:35:10.428603 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.428584 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:35:10.622607 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:10.622519 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" event={"ID":"47d7d674-b461-4be4-ab59-1beff66acbda","Type":"ContainerStarted","Data":"4e6c90919f31d85fa90be8455011ed907f5579d9383f8448a38c4484ae804dfe"} Jul 07 14:35:11.629368 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:11.629328 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" event={"ID":"47d7d674-b461-4be4-ab59-1beff66acbda","Type":"ContainerStarted","Data":"24089c09e588f11507043af3741f7a9521a0086f1dc2c6528dbe4be85db6c733"} Jul 07 14:35:11.652249 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:11.652192 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" podStartSLOduration=2.342156968 podStartE2EDuration="2.652172049s" totalImagesPullingTime="310.015081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:35:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:35:11.650975443 +0000 UTC m=+635.460523008" watchObservedRunningTime="2026-07-07 14:35:11.652172049 +0000 UTC m=+635.461719543" Jul 07 14:35:15.830938 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:15.830898 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:20.283895 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:20.283861 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:20.284284 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:20.283929 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:20.296494 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:20.296465 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:20.683151 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:20.683117 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w" Jul 07 14:35:24.662642 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.662605 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5"] Jul 07 14:35:24.668705 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.668681 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:24.671884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.671862 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 07 14:35:24.677260 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.677238 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5"] Jul 07 14:35:24.745008 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.744972 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/495d8e2e-1387-4209-8321-5e3233fb8f52-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5\" (UID: \"495d8e2e-1387-4209-8321-5e3233fb8f52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:24.745008 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.745010 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/495d8e2e-1387-4209-8321-5e3233fb8f52-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5\" (UID: \"495d8e2e-1387-4209-8321-5e3233fb8f52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:24.745222 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.745031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/495d8e2e-1387-4209-8321-5e3233fb8f52-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5\" (UID: \"495d8e2e-1387-4209-8321-5e3233fb8f52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:24.745222 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.745116 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495d8e2e-1387-4209-8321-5e3233fb8f52-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5\" (UID: \"495d8e2e-1387-4209-8321-5e3233fb8f52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:24.745222 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.745154 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495d8e2e-1387-4209-8321-5e3233fb8f52-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5\" (UID: \"495d8e2e-1387-4209-8321-5e3233fb8f52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:24.745222 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.745172 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jqww\" (UniqueName: \"kubernetes.io/projected/495d8e2e-1387-4209-8321-5e3233fb8f52-kube-api-access-2jqww\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5\" (UID: \"495d8e2e-1387-4209-8321-5e3233fb8f52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:24.762912 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.762880 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:24.846172 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.846135 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495d8e2e-1387-4209-8321-5e3233fb8f52-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5\" (UID: \"495d8e2e-1387-4209-8321-5e3233fb8f52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:24.846405 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.846189 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jqww\" (UniqueName: \"kubernetes.io/projected/495d8e2e-1387-4209-8321-5e3233fb8f52-kube-api-access-2jqww\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5\" (UID: \"495d8e2e-1387-4209-8321-5e3233fb8f52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:24.846405 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.846285 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/495d8e2e-1387-4209-8321-5e3233fb8f52-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5\" (UID: \"495d8e2e-1387-4209-8321-5e3233fb8f52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:24.846405 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.846308 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/495d8e2e-1387-4209-8321-5e3233fb8f52-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5\" (UID: \"495d8e2e-1387-4209-8321-5e3233fb8f52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:24.846405 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.846331 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/495d8e2e-1387-4209-8321-5e3233fb8f52-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5\" (UID: \"495d8e2e-1387-4209-8321-5e3233fb8f52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:24.846640 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.846464 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495d8e2e-1387-4209-8321-5e3233fb8f52-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5\" (UID: \"495d8e2e-1387-4209-8321-5e3233fb8f52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:24.846640 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.846597 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/495d8e2e-1387-4209-8321-5e3233fb8f52-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5\" (UID: \"495d8e2e-1387-4209-8321-5e3233fb8f52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:24.846705 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.846672 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/495d8e2e-1387-4209-8321-5e3233fb8f52-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5\" (UID: \"495d8e2e-1387-4209-8321-5e3233fb8f52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:24.846869 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.846833 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/495d8e2e-1387-4209-8321-5e3233fb8f52-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5\" (UID: \"495d8e2e-1387-4209-8321-5e3233fb8f52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:24.848914 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.848885 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/495d8e2e-1387-4209-8321-5e3233fb8f52-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5\" (UID: \"495d8e2e-1387-4209-8321-5e3233fb8f52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:24.849058 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.849039 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/495d8e2e-1387-4209-8321-5e3233fb8f52-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5\" (UID: \"495d8e2e-1387-4209-8321-5e3233fb8f52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:24.856773 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.856743 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jqww\" (UniqueName: \"kubernetes.io/projected/495d8e2e-1387-4209-8321-5e3233fb8f52-kube-api-access-2jqww\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5\" (UID: \"495d8e2e-1387-4209-8321-5e3233fb8f52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:24.979202 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:24.979111 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:25.121470 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:25.121425 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5"] Jul 07 14:35:25.125366 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:35:25.125337 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod495d8e2e_1387_4209_8321_5e3233fb8f52.slice/crio-c5600978732ef8d116cd88c61df135ea7921729d1222cf81255b6ba9d4c4e5cd WatchSource:0}: Error finding container c5600978732ef8d116cd88c61df135ea7921729d1222cf81255b6ba9d4c4e5cd: Status 404 returned error can't find the container with id c5600978732ef8d116cd88c61df135ea7921729d1222cf81255b6ba9d4c4e5cd Jul 07 14:35:25.706471 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:25.706433 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" event={"ID":"495d8e2e-1387-4209-8321-5e3233fb8f52","Type":"ContainerStarted","Data":"860c64714b0dcb59564dc9be5883fc2504eacd1645e233cd0ddf18332c9afb92"} Jul 07 14:35:25.706471 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:25.706477 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" event={"ID":"495d8e2e-1387-4209-8321-5e3233fb8f52","Type":"ContainerStarted","Data":"c5600978732ef8d116cd88c61df135ea7921729d1222cf81255b6ba9d4c4e5cd"} Jul 07 14:35:25.727831 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:25.727779 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" podStartSLOduration=1.498841543 podStartE2EDuration="1.727761562s" totalImagesPullingTime="228.920019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:35:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:35:25.726891226 +0000 UTC m=+649.536438728" watchObservedRunningTime="2026-07-07 14:35:25.727761562 +0000 UTC m=+649.537309053" Jul 07 14:35:26.320964 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:26.320929 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:34.979654 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:34.979614 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:34.980096 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:34.979673 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:34.992289 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:34.992261 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:35.772997 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:35.772970 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5" Jul 07 14:35:58.830002 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:58.829965 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:35:58.903608 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:35:58.903571 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:15.306275 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:15.306237 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:15.407302 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:15.407262 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:32.551878 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:32.551782 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8486594fb6-4nw5x"] Jul 07 14:36:32.555036 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:32.555020 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8486594fb6-4nw5x" Jul 07 14:36:32.568329 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:32.568303 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8486594fb6-4nw5x"] Jul 07 14:36:32.666655 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:32.666618 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w55jf\" (UniqueName: \"kubernetes.io/projected/b2e412e5-2397-4f98-8b6a-b8eb08bb77a4-kube-api-access-w55jf\") pod \"authorino-8486594fb6-4nw5x\" (UID: \"b2e412e5-2397-4f98-8b6a-b8eb08bb77a4\") " pod="kuadrant-system/authorino-8486594fb6-4nw5x" Jul 07 14:36:32.666841 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:32.666696 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b2e412e5-2397-4f98-8b6a-b8eb08bb77a4-tls-cert\") pod \"authorino-8486594fb6-4nw5x\" (UID: \"b2e412e5-2397-4f98-8b6a-b8eb08bb77a4\") " pod="kuadrant-system/authorino-8486594fb6-4nw5x" Jul 07 14:36:32.767608 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:32.767577 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w55jf\" (UniqueName: \"kubernetes.io/projected/b2e412e5-2397-4f98-8b6a-b8eb08bb77a4-kube-api-access-w55jf\") pod \"authorino-8486594fb6-4nw5x\" (UID: \"b2e412e5-2397-4f98-8b6a-b8eb08bb77a4\") " pod="kuadrant-system/authorino-8486594fb6-4nw5x" Jul 07 14:36:32.767795 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:32.767674 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b2e412e5-2397-4f98-8b6a-b8eb08bb77a4-tls-cert\") pod \"authorino-8486594fb6-4nw5x\" (UID: \"b2e412e5-2397-4f98-8b6a-b8eb08bb77a4\") " pod="kuadrant-system/authorino-8486594fb6-4nw5x" Jul 07 14:36:32.770259 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:32.770237 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b2e412e5-2397-4f98-8b6a-b8eb08bb77a4-tls-cert\") pod \"authorino-8486594fb6-4nw5x\" (UID: \"b2e412e5-2397-4f98-8b6a-b8eb08bb77a4\") " pod="kuadrant-system/authorino-8486594fb6-4nw5x" Jul 07 14:36:32.782096 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:32.782068 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w55jf\" (UniqueName: \"kubernetes.io/projected/b2e412e5-2397-4f98-8b6a-b8eb08bb77a4-kube-api-access-w55jf\") pod \"authorino-8486594fb6-4nw5x\" (UID: \"b2e412e5-2397-4f98-8b6a-b8eb08bb77a4\") " pod="kuadrant-system/authorino-8486594fb6-4nw5x" Jul 07 14:36:32.845280 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:32.845177 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:32.864648 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:32.864613 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8486594fb6-4nw5x" Jul 07 14:36:32.933796 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:32.933758 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:33.004445 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:33.004420 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8486594fb6-4nw5x"] Jul 07 14:36:33.006714 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:36:33.006682 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2e412e5_2397_4f98_8b6a_b8eb08bb77a4.slice/crio-5ebfa3a8e7ae6b35b2b373c9e3f3fc084daad4d5facb6a183309dfbdb1d5be99 WatchSource:0}: Error finding container 5ebfa3a8e7ae6b35b2b373c9e3f3fc084daad4d5facb6a183309dfbdb1d5be99: Status 404 returned error can't find the container with id 5ebfa3a8e7ae6b35b2b373c9e3f3fc084daad4d5facb6a183309dfbdb1d5be99 Jul 07 14:36:33.038020 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:33.037988 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8486594fb6-4nw5x" event={"ID":"b2e412e5-2397-4f98-8b6a-b8eb08bb77a4","Type":"ContainerStarted","Data":"5ebfa3a8e7ae6b35b2b373c9e3f3fc084daad4d5facb6a183309dfbdb1d5be99"} Jul 07 14:36:34.046346 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:34.046302 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8486594fb6-4nw5x" event={"ID":"b2e412e5-2397-4f98-8b6a-b8eb08bb77a4","Type":"ContainerStarted","Data":"89ded0582a6cc58d05b541bcf0adcaf029de89086d4747f251d7471690d7f46d"} Jul 07 14:36:34.062562 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:34.062521 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:34.083297 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:34.083229 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8486594fb6-4nw5x" podStartSLOduration=1.719079549 podStartE2EDuration="2.083203492s" totalImagesPullingTime="364.123943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:36:34.082734196 +0000 UTC m=+717.892281687" watchObservedRunningTime="2026-07-07 14:36:34.083203492 +0000 UTC m=+717.892750985" Jul 07 14:36:34.122986 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:34.122911 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c4d8c867-62hsh"] Jul 07 14:36:34.123267 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:34.123188 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5c4d8c867-62hsh" podUID="a3234c84-f3bc-4ea6-a97a-676e1aec05a4" containerName="authorino" containerID="cri-o://8531e0563b099f917f99b3ebd131aff966ed3fa3ac2bd0a5ce9b5ab7e25183ce" gracePeriod=30 Jul 07 14:36:34.153518 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:34.153483 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:36:34.383990 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:34.383959 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c4d8c867-62hsh" Jul 07 14:36:34.483459 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:34.483430 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhh27\" (UniqueName: \"kubernetes.io/projected/a3234c84-f3bc-4ea6-a97a-676e1aec05a4-kube-api-access-hhh27\") pod \"a3234c84-f3bc-4ea6-a97a-676e1aec05a4\" (UID: \"a3234c84-f3bc-4ea6-a97a-676e1aec05a4\") " Jul 07 14:36:34.483639 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:34.483545 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a3234c84-f3bc-4ea6-a97a-676e1aec05a4-tls-cert\") pod \"a3234c84-f3bc-4ea6-a97a-676e1aec05a4\" (UID: \"a3234c84-f3bc-4ea6-a97a-676e1aec05a4\") " Jul 07 14:36:34.485665 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:34.485634 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3234c84-f3bc-4ea6-a97a-676e1aec05a4-kube-api-access-hhh27" (OuterVolumeSpecName: "kube-api-access-hhh27") pod "a3234c84-f3bc-4ea6-a97a-676e1aec05a4" (UID: "a3234c84-f3bc-4ea6-a97a-676e1aec05a4"). InnerVolumeSpecName "kube-api-access-hhh27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:36:34.495108 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:34.495069 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3234c84-f3bc-4ea6-a97a-676e1aec05a4-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a3234c84-f3bc-4ea6-a97a-676e1aec05a4" (UID: "a3234c84-f3bc-4ea6-a97a-676e1aec05a4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:36:34.585364 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:34.585271 2562 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a3234c84-f3bc-4ea6-a97a-676e1aec05a4-tls-cert\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:36:34.585364 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:34.585306 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhh27\" (UniqueName: \"kubernetes.io/projected/a3234c84-f3bc-4ea6-a97a-676e1aec05a4-kube-api-access-hhh27\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:36:35.053411 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:35.053359 2562 generic.go:356] "Generic (PLEG): container finished" podID="a3234c84-f3bc-4ea6-a97a-676e1aec05a4" containerID="8531e0563b099f917f99b3ebd131aff966ed3fa3ac2bd0a5ce9b5ab7e25183ce" exitCode=0 Jul 07 14:36:35.053898 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:35.053423 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5c4d8c867-62hsh" Jul 07 14:36:35.053898 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:35.053448 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c4d8c867-62hsh" event={"ID":"a3234c84-f3bc-4ea6-a97a-676e1aec05a4","Type":"ContainerDied","Data":"8531e0563b099f917f99b3ebd131aff966ed3fa3ac2bd0a5ce9b5ab7e25183ce"} Jul 07 14:36:35.053898 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:35.053502 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5c4d8c867-62hsh" event={"ID":"a3234c84-f3bc-4ea6-a97a-676e1aec05a4","Type":"ContainerDied","Data":"89a1f798d6eea164166231fc8806148b6f99d9ee00e8deb3c1d63e07f7f39be6"} Jul 07 14:36:35.053898 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:35.053528 2562 scope.go:117] "RemoveContainer" containerID="8531e0563b099f917f99b3ebd131aff966ed3fa3ac2bd0a5ce9b5ab7e25183ce" Jul 07 14:36:35.063780 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:35.063750 2562 scope.go:117] "RemoveContainer" containerID="8531e0563b099f917f99b3ebd131aff966ed3fa3ac2bd0a5ce9b5ab7e25183ce" Jul 07 14:36:35.064119 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:36:35.064096 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8531e0563b099f917f99b3ebd131aff966ed3fa3ac2bd0a5ce9b5ab7e25183ce\": container with ID starting with 8531e0563b099f917f99b3ebd131aff966ed3fa3ac2bd0a5ce9b5ab7e25183ce not found: ID does not exist" containerID="8531e0563b099f917f99b3ebd131aff966ed3fa3ac2bd0a5ce9b5ab7e25183ce" Jul 07 14:36:35.064256 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:35.064126 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8531e0563b099f917f99b3ebd131aff966ed3fa3ac2bd0a5ce9b5ab7e25183ce"} err="failed to get container status \"8531e0563b099f917f99b3ebd131aff966ed3fa3ac2bd0a5ce9b5ab7e25183ce\": rpc error: code = NotFound desc = could not find container \"8531e0563b099f917f99b3ebd131aff966ed3fa3ac2bd0a5ce9b5ab7e25183ce\": container with ID starting with 8531e0563b099f917f99b3ebd131aff966ed3fa3ac2bd0a5ce9b5ab7e25183ce not found: ID does not exist" Jul 07 14:36:35.075750 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:35.075719 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5c4d8c867-62hsh"] Jul 07 14:36:35.081427 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:35.081402 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5c4d8c867-62hsh"] Jul 07 14:36:36.707191 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:36:36.707160 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3234c84-f3bc-4ea6-a97a-676e1aec05a4" path="/var/lib/kubelet/pods/a3234c84-f3bc-4ea6-a97a-676e1aec05a4/volumes" Jul 07 14:37:19.585588 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:19.585547 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:37:19.653495 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:19.653457 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:37:46.576079 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:46.576030 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:37:46.708392 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:46.708336 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6495c9587b-n2vbf"] Jul 07 14:37:46.708646 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:46.708580 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-6495c9587b-n2vbf" podUID="01c9d772-39f6-4c25-bdeb-944ccf78627b" containerName="manager" containerID="cri-o://8f0569a04392d669e215b87d7d29b0df40b51a239089cdd9c79cf2d28b30fc95" gracePeriod=30 Jul 07 14:37:46.959405 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:46.959358 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6495c9587b-n2vbf" Jul 07 14:37:47.086041 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:47.086000 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01c9d772-39f6-4c25-bdeb-944ccf78627b-webhook-cert\") pod \"01c9d772-39f6-4c25-bdeb-944ccf78627b\" (UID: \"01c9d772-39f6-4c25-bdeb-944ccf78627b\") " Jul 07 14:37:47.086176 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:47.086121 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnxc9\" (UniqueName: \"kubernetes.io/projected/01c9d772-39f6-4c25-bdeb-944ccf78627b-kube-api-access-wnxc9\") pod \"01c9d772-39f6-4c25-bdeb-944ccf78627b\" (UID: \"01c9d772-39f6-4c25-bdeb-944ccf78627b\") " Jul 07 14:37:47.088075 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:47.088041 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c9d772-39f6-4c25-bdeb-944ccf78627b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "01c9d772-39f6-4c25-bdeb-944ccf78627b" (UID: "01c9d772-39f6-4c25-bdeb-944ccf78627b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:37:47.088228 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:47.088098 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c9d772-39f6-4c25-bdeb-944ccf78627b-kube-api-access-wnxc9" (OuterVolumeSpecName: "kube-api-access-wnxc9") pod "01c9d772-39f6-4c25-bdeb-944ccf78627b" (UID: "01c9d772-39f6-4c25-bdeb-944ccf78627b"). InnerVolumeSpecName "kube-api-access-wnxc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:37:47.187515 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:47.187481 2562 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01c9d772-39f6-4c25-bdeb-944ccf78627b-webhook-cert\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:37:47.187515 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:47.187508 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnxc9\" (UniqueName: \"kubernetes.io/projected/01c9d772-39f6-4c25-bdeb-944ccf78627b-kube-api-access-wnxc9\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:37:47.405723 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:47.405685 2562 generic.go:356] "Generic (PLEG): container finished" podID="01c9d772-39f6-4c25-bdeb-944ccf78627b" containerID="8f0569a04392d669e215b87d7d29b0df40b51a239089cdd9c79cf2d28b30fc95" exitCode=0 Jul 07 14:37:47.405898 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:47.405737 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6495c9587b-n2vbf" Jul 07 14:37:47.405898 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:47.405773 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6495c9587b-n2vbf" event={"ID":"01c9d772-39f6-4c25-bdeb-944ccf78627b","Type":"ContainerDied","Data":"8f0569a04392d669e215b87d7d29b0df40b51a239089cdd9c79cf2d28b30fc95"} Jul 07 14:37:47.405898 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:47.405861 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6495c9587b-n2vbf" event={"ID":"01c9d772-39f6-4c25-bdeb-944ccf78627b","Type":"ContainerDied","Data":"2360dd22940a41d50e24e76c16eea870f1c0afc92c3b4fbe482471cf6cf1c249"} Jul 07 14:37:47.405898 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:47.405886 2562 scope.go:117] "RemoveContainer" containerID="8f0569a04392d669e215b87d7d29b0df40b51a239089cdd9c79cf2d28b30fc95" Jul 07 14:37:47.416364 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:47.416347 2562 scope.go:117] "RemoveContainer" containerID="8f0569a04392d669e215b87d7d29b0df40b51a239089cdd9c79cf2d28b30fc95" Jul 07 14:37:47.416622 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:37:47.416606 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0569a04392d669e215b87d7d29b0df40b51a239089cdd9c79cf2d28b30fc95\": container with ID starting with 8f0569a04392d669e215b87d7d29b0df40b51a239089cdd9c79cf2d28b30fc95 not found: ID does not exist" containerID="8f0569a04392d669e215b87d7d29b0df40b51a239089cdd9c79cf2d28b30fc95" Jul 07 14:37:47.416689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:47.416631 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0569a04392d669e215b87d7d29b0df40b51a239089cdd9c79cf2d28b30fc95"} err="failed to get container status \"8f0569a04392d669e215b87d7d29b0df40b51a239089cdd9c79cf2d28b30fc95\": rpc error: code = NotFound desc = could not find container \"8f0569a04392d669e215b87d7d29b0df40b51a239089cdd9c79cf2d28b30fc95\": container with ID starting with 8f0569a04392d669e215b87d7d29b0df40b51a239089cdd9c79cf2d28b30fc95 not found: ID does not exist" Jul 07 14:37:47.433884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:47.433853 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6495c9587b-n2vbf"] Jul 07 14:37:47.439808 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:47.439748 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6495c9587b-n2vbf"] Jul 07 14:37:48.139198 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:48.139167 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:37:48.709348 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:37:48.709317 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01c9d772-39f6-4c25-bdeb-944ccf78627b" path="/var/lib/kubelet/pods/01c9d772-39f6-4c25-bdeb-944ccf78627b/volumes" Jul 07 14:38:29.664674 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:38:29.664636 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:29.761021 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:38:29.760980 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:29.876989 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:38:29.876952 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:36.711775 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:38:36.711737 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:36.812038 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:38:36.811996 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:36.913918 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:38:36.913885 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:37.068286 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:38:37.068188 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:43.709496 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:38:43.709460 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:38:43.806475 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:38:43.806436 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:45.295418 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:39:45.295356 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:39:45.396254 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:39:45.396218 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:02.129983 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:42:02.129942 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:15.543236 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:42:15.543175 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:15.683851 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:42:15.683818 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:15.784922 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:42:15.784890 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:16.293626 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:42:16.293585 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:16.392171 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:42:16.392130 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:26.911815 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:42:26.911778 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:42:27.016030 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:42:27.015995 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:44:50.241539 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:44:50.241509 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:46:57.651735 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:46:57.651698 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:48:45.303592 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:48:45.303493 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:25.208816 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:49:25.208780 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:32.458692 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:49:32.458656 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:40.684626 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:49:40.684589 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:45.798888 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:49:45.798853 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:48.637790 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:49:48.637746 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:48.731171 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:49:48.731127 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:49.117027 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:49:49.116994 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:55.484266 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:49:55.484232 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:55.553798 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:49:55.553763 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:55.714799 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:49:55.714762 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:55.785268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:49:55.785181 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:56.903808 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:49:56.903769 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:57.010268 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:49:57.010236 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:57.115094 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:49:57.115057 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:57.223423 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:49:57.223316 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:57.695750 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:49:57.695717 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:57.793740 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:49:57.793701 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:49:59.306205 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:49:59.306162 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:02.089746 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:02.089711 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:02.260412 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:02.260359 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:02.707902 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:02.707865 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:02.896410 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:02.896343 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:04.280564 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:04.280519 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:08.587249 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:08.587208 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:08.694779 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:08.694729 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:08.837811 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:08.837725 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:08.907342 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:08.907283 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:09.991557 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:09.991524 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:10.072437 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:10.072398 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:10.146087 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:10.146055 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:10.430948 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:10.430915 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:10.712766 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:10.712649 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:10.812862 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:10.812823 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:12.208967 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:12.208918 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:15.101042 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:15.101003 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:15.629313 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:15.629278 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:21.365727 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:21.365690 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:21.487422 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:21.487385 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:23.163609 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:23.163570 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:28.815543 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:28.815502 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:28.920543 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:28.920504 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:29.105111 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:29.105032 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:29.206815 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:29.206778 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:30.272041 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:30.271991 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:30.386013 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:30.385980 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:30.526856 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:30.526765 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:30.654279 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:30.654245 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:31.073260 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:31.073226 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:31.173881 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:31.173836 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:32.481500 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:32.481465 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:34.717879 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:34.717847 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:34.867397 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:34.867351 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:34.981888 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:34.981788 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:35.322879 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:35.322793 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:36.204058 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:36.204026 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:42.352072 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:42.352034 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:42.485322 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:42.485286 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:42.894448 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:42.894408 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:43.008857 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:43.008819 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:43.095240 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:43.095200 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:44.581170 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:44.581118 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:51.290140 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:51.290103 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:51.359463 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:51.359423 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:51.448344 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:51.448313 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:51.549190 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:51.549105 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:51.649127 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:51.649093 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:50:52.062349 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:50:52.062309 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:51:18.125038 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:51:18.124995 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:51:26.382785 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:51:26.382749 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:51:26.511090 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:51:26.511040 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:51:26.626738 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:51:26.626698 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:51:42.531473 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:51:42.531381 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:51:49.100623 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:51:49.100583 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:51:49.197916 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:51:49.197880 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:51:49.300273 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:51:49.300238 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:51:59.859429 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:51:59.859368 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:00.694047 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:00.694009 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:07.128574 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:07.128527 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:07.232726 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:07.232691 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:07.358828 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:07.358790 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:07.456016 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:07.455980 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:08.239291 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:08.239256 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:08.435747 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:08.435712 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:10.082521 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:10.082485 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:16.962064 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:16.962025 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:17.080401 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:17.080342 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:17.734586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:17.734539 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:24.438327 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:24.438289 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:24.538002 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:24.537968 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:24.704229 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:24.704150 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:25.051950 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:25.051848 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:25.605644 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:25.605609 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:28.413986 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:28.413943 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:28.513016 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:28.512956 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:28.610401 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:28.610336 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:28.948321 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:28.948280 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:29.343580 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:29.343491 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:29.459205 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:29.459158 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:32.933971 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:32.933931 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:33.607942 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:33.607902 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:33.709554 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:33.709517 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:34.094717 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:34.094683 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:40.540851 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:40.540813 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:40.641437 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:40.641401 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:40.959410 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:40.959358 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:41.375760 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:41.375675 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:41.473829 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:41.473797 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:47.781264 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:47.781225 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:47.882117 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:47.882073 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:48.074884 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:48.074783 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:48.305399 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:48.305339 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:48.413803 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:48.413772 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:48.515765 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:48.515730 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:48.650199 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:48.650167 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:55.184579 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:55.184535 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:55.292966 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:55.292931 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:55.385599 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:55.385563 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:52:55.550973 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:52:55.550887 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:02.066912 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:53:02.066822 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:02.133393 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:53:02.133342 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:08.965809 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:53:08.965760 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:09.401018 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:53:09.400981 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:09.552784 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:53:09.552745 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:10.094924 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:53:10.094889 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:10.195929 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:53:10.195892 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:12.700051 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:53:12.700012 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:12.795448 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:53:12.795408 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:12.875180 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:53:12.875144 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:13.420897 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:53:13.420865 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:13.520448 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:53:13.520410 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:13.620809 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:53:13.620772 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:17.261520 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:53:17.261483 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:53:34.189473 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:53:34.189436 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:54:05.500579 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:54:05.500541 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:54:06.698877 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:54:06.698840 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:54:11.782830 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:54:11.782795 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:54:16.826004 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:54:16.825967 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:54:53.561297 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:54:53.561254 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:54:54.818598 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:54:54.818560 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:00.960277 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:00.960237 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:01.060928 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:01.060889 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:01.368324 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:01.368229 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:01.583070 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:01.583025 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:02.666632 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:02.666590 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:02.765526 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:02.765488 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:02.865658 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:02.865627 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:03.686408 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:03.686365 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:03.781409 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:03.781351 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:03.875139 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:03.875104 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:05.446369 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:05.446327 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:12.093451 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:12.093413 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:12.187434 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:12.187399 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:12.766764 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:12.766723 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:13.181200 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:13.181159 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:13.281537 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:13.281499 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:19.150280 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:19.150238 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:19.218086 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:19.218049 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:19.312225 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:19.312186 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:19.411263 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:19.406333 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:29.901388 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:29.901346 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:30.266936 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:30.266845 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:51.274325 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:51.274287 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:57.829862 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:57.829826 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:57.930554 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:57.930510 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:58.183126 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:58.183083 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:58.505160 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:58.505078 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:55:58.625149 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:55:58.625116 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:01.498026 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:01.497981 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:01.601127 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:01.601096 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:01.702353 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:01.702317 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:01.797401 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:01.797290 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:02.090863 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:02.090779 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:02.187273 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:02.187241 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:06.030710 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:06.030673 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:06.419257 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:06.419216 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:06.821196 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:06.821108 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:06.982951 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:06.982912 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:10.042147 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:10.042112 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:10.165346 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:10.165315 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:10.270486 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:10.270450 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:10.442442 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:10.442404 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:10.761586 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:10.761497 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:10.859546 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:10.859481 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:14.630638 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:14.630592 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:20.217656 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:20.217616 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:23.016265 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:23.016220 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:23.112287 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:23.112251 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:32.631537 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:32.631500 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:32.731725 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:32.731695 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:35.249497 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.249414 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c"] Jul 07 14:56:35.249838 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.249812 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01c9d772-39f6-4c25-bdeb-944ccf78627b" containerName="manager" Jul 07 14:56:35.249838 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.249825 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="01c9d772-39f6-4c25-bdeb-944ccf78627b" containerName="manager" Jul 07 14:56:35.249911 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.249844 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3234c84-f3bc-4ea6-a97a-676e1aec05a4" containerName="authorino" Jul 07 14:56:35.249911 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.249851 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3234c84-f3bc-4ea6-a97a-676e1aec05a4" containerName="authorino" Jul 07 14:56:35.249975 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.249928 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="01c9d772-39f6-4c25-bdeb-944ccf78627b" containerName="manager" Jul 07 14:56:35.249975 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.249940 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3234c84-f3bc-4ea6-a97a-676e1aec05a4" containerName="authorino" Jul 07 14:56:35.253124 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.253105 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.256665 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.256636 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c289a3\"/\"default-dockercfg-pgr42\"" Jul 07 14:56:35.256665 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.256646 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c289a3\"/\"openshift-service-ca.crt\"" Jul 07 14:56:35.256665 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.256663 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c289a3\"/\"test-model-c289a3-kserve-self-signed-certs\"" Jul 07 14:56:35.256892 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.256715 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c289a3\"/\"kube-root-ca.crt\"" Jul 07 14:56:35.263918 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.263897 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c"] Jul 07 14:56:35.284402 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.284358 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-model-cache\") pod \"test-model-c289a3-kserve-579d75bc85-2jt8c\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.284600 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.284407 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbxc7\" (UniqueName: \"kubernetes.io/projected/e453d9bc-c2f7-4166-9368-bc118ac490b0-kube-api-access-mbxc7\") pod \"test-model-c289a3-kserve-579d75bc85-2jt8c\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.284600 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.284462 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-dshm\") pod \"test-model-c289a3-kserve-579d75bc85-2jt8c\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.284600 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.284489 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-tmp-dir\") pod \"test-model-c289a3-kserve-579d75bc85-2jt8c\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.284600 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.284572 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-home\") pod \"test-model-c289a3-kserve-579d75bc85-2jt8c\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.284807 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.284607 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e453d9bc-c2f7-4166-9368-bc118ac490b0-tls-certs\") pod \"test-model-c289a3-kserve-579d75bc85-2jt8c\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.299681 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.299637 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:56:35.385681 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.385643 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-home\") pod \"test-model-c289a3-kserve-579d75bc85-2jt8c\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.385865 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.385693 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e453d9bc-c2f7-4166-9368-bc118ac490b0-tls-certs\") pod \"test-model-c289a3-kserve-579d75bc85-2jt8c\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.385865 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.385741 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-model-cache\") pod \"test-model-c289a3-kserve-579d75bc85-2jt8c\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.385865 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.385764 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbxc7\" (UniqueName: \"kubernetes.io/projected/e453d9bc-c2f7-4166-9368-bc118ac490b0-kube-api-access-mbxc7\") pod \"test-model-c289a3-kserve-579d75bc85-2jt8c\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.385865 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.385825 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-dshm\") pod \"test-model-c289a3-kserve-579d75bc85-2jt8c\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.385865 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.385846 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-tmp-dir\") pod \"test-model-c289a3-kserve-579d75bc85-2jt8c\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.386155 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.386131 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-home\") pod \"test-model-c289a3-kserve-579d75bc85-2jt8c\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.386222 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.386181 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-model-cache\") pod \"test-model-c289a3-kserve-579d75bc85-2jt8c\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.386269 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.386235 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-tmp-dir\") pod \"test-model-c289a3-kserve-579d75bc85-2jt8c\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.388122 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.388101 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-dshm\") pod \"test-model-c289a3-kserve-579d75bc85-2jt8c\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.388284 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.388264 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e453d9bc-c2f7-4166-9368-bc118ac490b0-tls-certs\") pod \"test-model-c289a3-kserve-579d75bc85-2jt8c\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.397123 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.397091 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbxc7\" (UniqueName: \"kubernetes.io/projected/e453d9bc-c2f7-4166-9368-bc118ac490b0-kube-api-access-mbxc7\") pod \"test-model-c289a3-kserve-579d75bc85-2jt8c\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.562636 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.562537 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:35.710411 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.703943 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c"] Jul 07 14:56:35.711954 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:56:35.711926 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode453d9bc_c2f7_4166_9368_bc118ac490b0.slice/crio-e7752c875df21411449740107cbe1b34e574bef44d3a6966295d4ad5a1758c7c WatchSource:0}: Error finding container e7752c875df21411449740107cbe1b34e574bef44d3a6966295d4ad5a1758c7c: Status 404 returned error can't find the container with id e7752c875df21411449740107cbe1b34e574bef44d3a6966295d4ad5a1758c7c Jul 07 14:56:35.875829 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.875724 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" event={"ID":"e453d9bc-c2f7-4166-9368-bc118ac490b0","Type":"ContainerStarted","Data":"24ca9f98ca3874a96669c7c141837c94b6999da27d826383f948ffe05879ac20"} Jul 07 14:56:35.875829 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.875773 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" event={"ID":"e453d9bc-c2f7-4166-9368-bc118ac490b0","Type":"ContainerStarted","Data":"e7752c875df21411449740107cbe1b34e574bef44d3a6966295d4ad5a1758c7c"} Jul 07 14:56:35.898084 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:35.898027 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" podStartSLOduration=0.898010887 podStartE2EDuration="898.010887ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:56:35.89654022 +0000 UTC m=+1919.706087709" watchObservedRunningTime="2026-07-07 14:56:35.898010887 +0000 UTC m=+1919.707558377" Jul 07 14:56:45.563915 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:45.563870 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:45.564361 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:45.563936 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:45.576483 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:45.576455 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:56:45.934998 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:56:45.934969 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:57:16.573409 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.573361 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs"] Jul 07 14:57:16.577037 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.577018 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:16.580696 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.580670 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cf70a3\"/\"kube-root-ca.crt\"" Jul 07 14:57:16.580696 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.580683 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cf70a3\"/\"test-model-cf70a3-kserve-self-signed-certs\"" Jul 07 14:57:16.580881 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.580671 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cf70a3\"/\"openshift-service-ca.crt\"" Jul 07 14:57:16.580881 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.580680 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cf70a3\"/\"default-dockercfg-gchj9\"" Jul 07 14:57:16.588243 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.588212 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs"] Jul 07 14:57:16.601272 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.601221 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:57:16.678814 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.678776 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-dshm\") pod \"test-model-cf70a3-kserve-8997dcdb9-xcxbs\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:16.678998 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.678868 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-tls-certs\") pod \"test-model-cf70a3-kserve-8997dcdb9-xcxbs\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:16.678998 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.678916 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-model-cache\") pod \"test-model-cf70a3-kserve-8997dcdb9-xcxbs\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:16.678998 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.678935 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-home\") pod \"test-model-cf70a3-kserve-8997dcdb9-xcxbs\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:16.678998 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.678958 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-tmp-dir\") pod \"test-model-cf70a3-kserve-8997dcdb9-xcxbs\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:16.679150 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.679023 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pplz\" (UniqueName: \"kubernetes.io/projected/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-kube-api-access-6pplz\") pod \"test-model-cf70a3-kserve-8997dcdb9-xcxbs\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:16.779729 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.779693 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-tls-certs\") pod \"test-model-cf70a3-kserve-8997dcdb9-xcxbs\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:16.779935 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.779739 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-model-cache\") pod \"test-model-cf70a3-kserve-8997dcdb9-xcxbs\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:16.779935 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.779763 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-home\") pod \"test-model-cf70a3-kserve-8997dcdb9-xcxbs\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:16.779935 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.779786 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-tmp-dir\") pod \"test-model-cf70a3-kserve-8997dcdb9-xcxbs\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:16.779935 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.779852 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pplz\" (UniqueName: \"kubernetes.io/projected/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-kube-api-access-6pplz\") pod \"test-model-cf70a3-kserve-8997dcdb9-xcxbs\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:16.779935 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.779880 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-dshm\") pod \"test-model-cf70a3-kserve-8997dcdb9-xcxbs\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:16.780299 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.780176 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-model-cache\") pod \"test-model-cf70a3-kserve-8997dcdb9-xcxbs\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:16.780299 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.780263 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-tmp-dir\") pod \"test-model-cf70a3-kserve-8997dcdb9-xcxbs\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:16.780299 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.780291 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-home\") pod \"test-model-cf70a3-kserve-8997dcdb9-xcxbs\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:16.782151 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.782127 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-dshm\") pod \"test-model-cf70a3-kserve-8997dcdb9-xcxbs\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:16.782360 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.782341 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-tls-certs\") pod \"test-model-cf70a3-kserve-8997dcdb9-xcxbs\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:16.790705 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.790674 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pplz\" (UniqueName: \"kubernetes.io/projected/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-kube-api-access-6pplz\") pod \"test-model-cf70a3-kserve-8997dcdb9-xcxbs\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:16.886800 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:16.886701 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:17.032471 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:17.032444 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs"] Jul 07 14:57:17.034812 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:57:17.034772 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba7f0968_84f5_47cf_b3cd_e35c6fcd6993.slice/crio-e94f896d273dbfb66452d56a5b33e16bae8a41b78959593d06c2970bc337b71d WatchSource:0}: Error finding container e94f896d273dbfb66452d56a5b33e16bae8a41b78959593d06c2970bc337b71d: Status 404 returned error can't find the container with id e94f896d273dbfb66452d56a5b33e16bae8a41b78959593d06c2970bc337b71d Jul 07 14:57:17.075518 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:17.075484 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" event={"ID":"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993","Type":"ContainerStarted","Data":"e94f896d273dbfb66452d56a5b33e16bae8a41b78959593d06c2970bc337b71d"} Jul 07 14:57:18.081923 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:18.081882 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" event={"ID":"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993","Type":"ContainerStarted","Data":"702b9917d02c92b8545708d528e1f5a039b53a2b92fb367717b9a9ae8ca556ed"} Jul 07 14:57:18.104034 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:18.103981 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" podStartSLOduration=2.103963256 podStartE2EDuration="2.103963256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:57:18.103360856 +0000 UTC m=+1961.912908358" watchObservedRunningTime="2026-07-07 14:57:18.103963256 +0000 UTC m=+1961.913510747" Jul 07 14:57:26.887068 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:26.887025 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:26.887761 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:26.887092 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:26.899501 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:26.899474 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:27.138026 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:27.137946 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:39.014734 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:39.014695 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c"] Jul 07 14:57:39.021284 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:39.021238 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:57:40.099986 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:40.099954 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs"] Jul 07 14:57:40.111612 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:40.111561 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:57:40.320741 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:40.320703 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:57:40.654399 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:40.654337 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:57:40.968245 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:40.968154 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:57:41.089430 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:41.089394 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:57:41.310819 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:41.310709 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:57:45.810734 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:45.810693 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:57:54.035003 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.034936 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" podUID="e453d9bc-c2f7-4166-9368-bc118ac490b0" containerName="main" containerID="cri-o://24ca9f98ca3874a96669c7c141837c94b6999da27d826383f948ffe05879ac20" gracePeriod=45 Jul 07 14:57:54.260865 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.260828 2562 generic.go:356] "Generic (PLEG): container finished" podID="e453d9bc-c2f7-4166-9368-bc118ac490b0" containerID="24ca9f98ca3874a96669c7c141837c94b6999da27d826383f948ffe05879ac20" exitCode=0 Jul 07 14:57:54.261028 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.260900 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" event={"ID":"e453d9bc-c2f7-4166-9368-bc118ac490b0","Type":"ContainerDied","Data":"24ca9f98ca3874a96669c7c141837c94b6999da27d826383f948ffe05879ac20"} Jul 07 14:57:54.295387 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.295308 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:57:54.341894 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.341861 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-dshm\") pod \"e453d9bc-c2f7-4166-9368-bc118ac490b0\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " Jul 07 14:57:54.342059 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.341916 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-tmp-dir\") pod \"e453d9bc-c2f7-4166-9368-bc118ac490b0\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " Jul 07 14:57:54.342059 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.341935 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-model-cache\") pod \"e453d9bc-c2f7-4166-9368-bc118ac490b0\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " Jul 07 14:57:54.342059 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.341952 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e453d9bc-c2f7-4166-9368-bc118ac490b0-tls-certs\") pod \"e453d9bc-c2f7-4166-9368-bc118ac490b0\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " Jul 07 14:57:54.342059 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.341989 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbxc7\" (UniqueName: \"kubernetes.io/projected/e453d9bc-c2f7-4166-9368-bc118ac490b0-kube-api-access-mbxc7\") pod \"e453d9bc-c2f7-4166-9368-bc118ac490b0\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " Jul 07 14:57:54.342059 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.342012 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-home\") pod \"e453d9bc-c2f7-4166-9368-bc118ac490b0\" (UID: \"e453d9bc-c2f7-4166-9368-bc118ac490b0\") " Jul 07 14:57:54.342351 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.342164 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e453d9bc-c2f7-4166-9368-bc118ac490b0" (UID: "e453d9bc-c2f7-4166-9368-bc118ac490b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:57:54.342351 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.342262 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "e453d9bc-c2f7-4166-9368-bc118ac490b0" (UID: "e453d9bc-c2f7-4166-9368-bc118ac490b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:57:54.342351 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.342336 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-home" (OuterVolumeSpecName: "home") pod "e453d9bc-c2f7-4166-9368-bc118ac490b0" (UID: "e453d9bc-c2f7-4166-9368-bc118ac490b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:57:54.342525 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.342484 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-tmp-dir\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:57:54.342525 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.342499 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-model-cache\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:57:54.342525 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.342509 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-home\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:57:54.344235 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.344206 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-dshm" (OuterVolumeSpecName: "dshm") pod "e453d9bc-c2f7-4166-9368-bc118ac490b0" (UID: "e453d9bc-c2f7-4166-9368-bc118ac490b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:57:54.344385 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.344349 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e453d9bc-c2f7-4166-9368-bc118ac490b0-kube-api-access-mbxc7" (OuterVolumeSpecName: "kube-api-access-mbxc7") pod "e453d9bc-c2f7-4166-9368-bc118ac490b0" (UID: "e453d9bc-c2f7-4166-9368-bc118ac490b0"). InnerVolumeSpecName "kube-api-access-mbxc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:57:54.344488 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.344469 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e453d9bc-c2f7-4166-9368-bc118ac490b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e453d9bc-c2f7-4166-9368-bc118ac490b0" (UID: "e453d9bc-c2f7-4166-9368-bc118ac490b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:57:54.443087 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.443037 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e453d9bc-c2f7-4166-9368-bc118ac490b0-dshm\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:57:54.443087 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.443080 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e453d9bc-c2f7-4166-9368-bc118ac490b0-tls-certs\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:57:54.443087 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:54.443091 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbxc7\" (UniqueName: \"kubernetes.io/projected/e453d9bc-c2f7-4166-9368-bc118ac490b0-kube-api-access-mbxc7\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:57:55.120476 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.120345 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" podUID="ba7f0968-84f5-47cf-b3cd-e35c6fcd6993" containerName="main" containerID="cri-o://702b9917d02c92b8545708d528e1f5a039b53a2b92fb367717b9a9ae8ca556ed" gracePeriod=45 Jul 07 14:57:55.268137 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.268100 2562 generic.go:356] "Generic (PLEG): container finished" podID="ba7f0968-84f5-47cf-b3cd-e35c6fcd6993" containerID="702b9917d02c92b8545708d528e1f5a039b53a2b92fb367717b9a9ae8ca556ed" exitCode=0 Jul 07 14:57:55.268313 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.268177 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" event={"ID":"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993","Type":"ContainerDied","Data":"702b9917d02c92b8545708d528e1f5a039b53a2b92fb367717b9a9ae8ca556ed"} Jul 07 14:57:55.269653 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.269627 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" event={"ID":"e453d9bc-c2f7-4166-9368-bc118ac490b0","Type":"ContainerDied","Data":"e7752c875df21411449740107cbe1b34e574bef44d3a6966295d4ad5a1758c7c"} Jul 07 14:57:55.269653 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.269648 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c" Jul 07 14:57:55.269882 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.269672 2562 scope.go:117] "RemoveContainer" containerID="24ca9f98ca3874a96669c7c141837c94b6999da27d826383f948ffe05879ac20" Jul 07 14:57:55.291513 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.291482 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c"] Jul 07 14:57:55.301907 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.301880 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c289a3/test-model-c289a3-kserve-579d75bc85-2jt8c"] Jul 07 14:57:55.385689 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.385626 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:55.451008 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.450973 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-home\") pod \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " Jul 07 14:57:55.451008 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.451017 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-tls-certs\") pod \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " Jul 07 14:57:55.451266 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.451033 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-model-cache\") pod \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " Jul 07 14:57:55.451266 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.451063 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pplz\" (UniqueName: \"kubernetes.io/projected/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-kube-api-access-6pplz\") pod \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " Jul 07 14:57:55.451266 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.451088 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-tmp-dir\") pod \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " Jul 07 14:57:55.451266 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.451172 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-dshm\") pod \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\" (UID: \"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993\") " Jul 07 14:57:55.451493 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.451346 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba7f0968-84f5-47cf-b3cd-e35c6fcd6993" (UID: "ba7f0968-84f5-47cf-b3cd-e35c6fcd6993"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:57:55.451493 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.451362 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-home" (OuterVolumeSpecName: "home") pod "ba7f0968-84f5-47cf-b3cd-e35c6fcd6993" (UID: "ba7f0968-84f5-47cf-b3cd-e35c6fcd6993"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:57:55.451493 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.451455 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba7f0968-84f5-47cf-b3cd-e35c6fcd6993" (UID: "ba7f0968-84f5-47cf-b3cd-e35c6fcd6993"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:57:55.451613 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.451516 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-home\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:57:55.451613 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.451535 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-model-cache\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:57:55.453433 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.453400 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba7f0968-84f5-47cf-b3cd-e35c6fcd6993" (UID: "ba7f0968-84f5-47cf-b3cd-e35c6fcd6993"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 14:57:55.453551 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.453429 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-kube-api-access-6pplz" (OuterVolumeSpecName: "kube-api-access-6pplz") pod "ba7f0968-84f5-47cf-b3cd-e35c6fcd6993" (UID: "ba7f0968-84f5-47cf-b3cd-e35c6fcd6993"). InnerVolumeSpecName "kube-api-access-6pplz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 14:57:55.453551 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.453488 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-dshm" (OuterVolumeSpecName: "dshm") pod "ba7f0968-84f5-47cf-b3cd-e35c6fcd6993" (UID: "ba7f0968-84f5-47cf-b3cd-e35c6fcd6993"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 14:57:55.552190 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.552147 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-tls-certs\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:57:55.552190 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.552183 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pplz\" (UniqueName: \"kubernetes.io/projected/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-kube-api-access-6pplz\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:57:55.552190 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.552194 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-tmp-dir\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:57:55.552472 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:55.552204 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993-dshm\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 14:57:56.276365 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:56.276333 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" event={"ID":"ba7f0968-84f5-47cf-b3cd-e35c6fcd6993","Type":"ContainerDied","Data":"e94f896d273dbfb66452d56a5b33e16bae8a41b78959593d06c2970bc337b71d"} Jul 07 14:57:56.276806 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:56.276388 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs" Jul 07 14:57:56.276806 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:56.276398 2562 scope.go:117] "RemoveContainer" containerID="702b9917d02c92b8545708d528e1f5a039b53a2b92fb367717b9a9ae8ca556ed" Jul 07 14:57:56.303122 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:56.303090 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs"] Jul 07 14:57:56.309095 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:56.309069 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cf70a3/test-model-cf70a3-kserve-8997dcdb9-xcxbs"] Jul 07 14:57:56.707864 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:56.707830 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba7f0968-84f5-47cf-b3cd-e35c6fcd6993" path="/var/lib/kubelet/pods/ba7f0968-84f5-47cf-b3cd-e35c6fcd6993/volumes" Jul 07 14:57:56.708234 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:57:56.708221 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e453d9bc-c2f7-4166-9368-bc118ac490b0" path="/var/lib/kubelet/pods/e453d9bc-c2f7-4166-9368-bc118ac490b0/volumes" Jul 07 14:58:01.453939 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:01.453902 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:01.556771 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:01.556733 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:11.113478 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:11.113441 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:11.213445 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:11.213413 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:23.543404 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:23.543350 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:23.830223 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:23.830137 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:24.041515 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:24.041486 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:24.260341 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:24.260296 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:24.648802 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:24.648767 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:24.912117 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:24.912012 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:25.010040 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:25.010002 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:25.167759 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:25.167670 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:26.860910 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:26.860870 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:31.729161 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:31.729122 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:31.828982 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:31.828947 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:32.039981 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:32.039878 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:32.250461 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:32.250422 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:32.583764 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:32.583726 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:32.933288 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:32.933250 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:33.091143 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:33.091104 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:36.322177 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:36.322132 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:36.426431 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:36.426392 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:36.538485 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:36.538454 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:37.132880 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:37.132790 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:37.230752 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:37.230715 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:40.809864 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:40.809823 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:41.400424 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:41.400388 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:41.690449 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:41.690329 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:41.851867 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:41.851829 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:44.396759 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:44.396713 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:44.497121 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:44.497082 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:44.615505 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:44.615465 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:45.132322 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:45.132290 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:45.231272 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:45.231233 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:48.802621 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:48.802587 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:57.019585 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:57.019546 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:57.464473 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:57.464434 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:57.611089 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:57.611045 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:57.711711 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:57.711677 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:58:57.913199 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:58:57.913162 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:04.147667 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:04.147629 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:04.244828 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:04.244787 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:04.422238 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:04.422150 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:04.649056 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:04.649011 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:04.848963 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:04.848870 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:05.242006 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:05.241965 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:05.412015 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:05.411981 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:05.499149 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:05.499059 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:05.720810 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:05.720780 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:11.652633 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:11.652600 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:13.123059 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:13.123017 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:15.892282 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:15.892237 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:15.975360 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:15.975330 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:16.057604 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:16.057573 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:16.172514 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:16.172476 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:16.601849 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:16.601751 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:16.699388 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:16.699352 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:20.362923 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:20.362890 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:20.706626 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:20.706590 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:21.031563 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:21.031446 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:21.183532 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:21.183496 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:23.528844 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.528787 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:23.605769 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.605739 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472"] Jul 07 14:59:23.606219 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.606201 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba7f0968-84f5-47cf-b3cd-e35c6fcd6993" containerName="main" Jul 07 14:59:23.606313 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.606229 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba7f0968-84f5-47cf-b3cd-e35c6fcd6993" containerName="main" Jul 07 14:59:23.606313 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.606264 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e453d9bc-c2f7-4166-9368-bc118ac490b0" containerName="main" Jul 07 14:59:23.606313 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.606273 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e453d9bc-c2f7-4166-9368-bc118ac490b0" containerName="main" Jul 07 14:59:23.606503 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.606419 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e453d9bc-c2f7-4166-9368-bc118ac490b0" containerName="main" Jul 07 14:59:23.606503 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.606437 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba7f0968-84f5-47cf-b3cd-e35c6fcd6993" containerName="main" Jul 07 14:59:23.609730 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.609709 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" Jul 07 14:59:23.612727 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.612701 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-537d76-serving-cert\"" Jul 07 14:59:23.619528 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.619486 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:23.634740 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.634706 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472"] Jul 07 14:59:23.671921 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.671889 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swx8x\" (UniqueName: \"kubernetes.io/projected/486a1e57-5fd6-446b-b0f8-85fa847b7489-kube-api-access-swx8x\") pod \"maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472\" (UID: \"486a1e57-5fd6-446b-b0f8-85fa847b7489\") " pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" Jul 07 14:59:23.672095 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.671933 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/486a1e57-5fd6-446b-b0f8-85fa847b7489-maas-api-tls\") pod \"maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472\" (UID: \"486a1e57-5fd6-446b-b0f8-85fa847b7489\") " pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" Jul 07 14:59:23.672095 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.672006 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/486a1e57-5fd6-446b-b0f8-85fa847b7489-openshift-service-ca\") pod \"maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472\" (UID: \"486a1e57-5fd6-446b-b0f8-85fa847b7489\") " pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" Jul 07 14:59:23.705116 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.705084 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:23.772843 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.772803 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swx8x\" (UniqueName: \"kubernetes.io/projected/486a1e57-5fd6-446b-b0f8-85fa847b7489-kube-api-access-swx8x\") pod \"maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472\" (UID: \"486a1e57-5fd6-446b-b0f8-85fa847b7489\") " pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" Jul 07 14:59:23.773030 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.772854 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/486a1e57-5fd6-446b-b0f8-85fa847b7489-maas-api-tls\") pod \"maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472\" (UID: \"486a1e57-5fd6-446b-b0f8-85fa847b7489\") " pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" Jul 07 14:59:23.773030 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.772883 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/486a1e57-5fd6-446b-b0f8-85fa847b7489-openshift-service-ca\") pod \"maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472\" (UID: \"486a1e57-5fd6-446b-b0f8-85fa847b7489\") " pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" Jul 07 14:59:23.773105 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:59:23.773022 2562 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-537d76-serving-cert: secret "maas-api-e2e-shared-b-537d76-serving-cert" not found Jul 07 14:59:23.773147 ip-10-0-128-34 kubenswrapper[2562]: E0707 14:59:23.773121 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/486a1e57-5fd6-446b-b0f8-85fa847b7489-maas-api-tls podName:486a1e57-5fd6-446b-b0f8-85fa847b7489 nodeName:}" failed. No retries permitted until 2026-07-07 14:59:24.2731028 +0000 UTC m=+2088.082650268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/486a1e57-5fd6-446b-b0f8-85fa847b7489-maas-api-tls") pod "maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" (UID: "486a1e57-5fd6-446b-b0f8-85fa847b7489") : secret "maas-api-e2e-shared-b-537d76-serving-cert" not found Jul 07 14:59:23.773582 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.773564 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/486a1e57-5fd6-446b-b0f8-85fa847b7489-openshift-service-ca\") pod \"maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472\" (UID: \"486a1e57-5fd6-446b-b0f8-85fa847b7489\") " pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" Jul 07 14:59:23.793327 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:23.793247 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swx8x\" (UniqueName: \"kubernetes.io/projected/486a1e57-5fd6-446b-b0f8-85fa847b7489-kube-api-access-swx8x\") pod \"maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472\" (UID: \"486a1e57-5fd6-446b-b0f8-85fa847b7489\") " pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" Jul 07 14:59:24.142404 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:24.142301 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:24.277818 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:24.277778 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/486a1e57-5fd6-446b-b0f8-85fa847b7489-maas-api-tls\") pod \"maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472\" (UID: \"486a1e57-5fd6-446b-b0f8-85fa847b7489\") " pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" Jul 07 14:59:24.280230 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:24.280206 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/486a1e57-5fd6-446b-b0f8-85fa847b7489-maas-api-tls\") pod \"maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472\" (UID: \"486a1e57-5fd6-446b-b0f8-85fa847b7489\") " pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" Jul 07 14:59:24.282981 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:24.282948 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:24.385367 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:24.385334 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:24.521730 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:24.521687 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" Jul 07 14:59:24.659967 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:24.659943 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472"] Jul 07 14:59:24.661861 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:59:24.661822 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486a1e57_5fd6_446b_b0f8_85fa847b7489.slice/crio-02ccdf561d397c3badb2f5728f54ac86edca2587debb73e658884911de94edf0 WatchSource:0}: Error finding container 02ccdf561d397c3badb2f5728f54ac86edca2587debb73e658884911de94edf0: Status 404 returned error can't find the container with id 02ccdf561d397c3badb2f5728f54ac86edca2587debb73e658884911de94edf0 Jul 07 14:59:24.663254 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:24.663234 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 07 14:59:24.709343 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:24.709310 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" event={"ID":"486a1e57-5fd6-446b-b0f8-85fa847b7489","Type":"ContainerStarted","Data":"02ccdf561d397c3badb2f5728f54ac86edca2587debb73e658884911de94edf0"} Jul 07 14:59:25.715746 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:25.715710 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" event={"ID":"486a1e57-5fd6-446b-b0f8-85fa847b7489","Type":"ContainerStarted","Data":"78c8ab0280d25a25561bd5e65b9f05c5e54eb7166b676c79c1a13e1fda21e358"} Jul 07 14:59:25.716160 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:25.715815 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" Jul 07 14:59:25.737899 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:25.737847 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" podStartSLOduration=2.441241586 podStartE2EDuration="2.737829508s" totalImagesPullingTime="296.587922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:59:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:59:25.735260492 +0000 UTC m=+2089.544807995" watchObservedRunningTime="2026-07-07 14:59:25.737829508 +0000 UTC m=+2089.547376997" Jul 07 14:59:28.101581 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:28.101539 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:31.725758 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:31.725728 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" Jul 07 14:59:36.855938 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:36.855906 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m"] Jul 07 14:59:36.859556 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:36.859536 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:36.862101 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:36.862072 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-82db1a\"/\"openshift-service-ca.crt\"" Jul 07 14:59:36.862254 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:36.862141 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-82db1a\"/\"kube-root-ca.crt\"" Jul 07 14:59:36.862254 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:36.862226 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-82db1a\"/\"default-dockercfg-x5scf\"" Jul 07 14:59:36.863024 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:36.863001 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-82db1a\"/\"auth-test-model-82db1a-kserve-self-signed-certs\"" Jul 07 14:59:36.869607 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:36.869581 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m"] Jul 07 14:59:36.886548 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:36.886516 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:37.001633 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.001597 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-tmp-dir\") pod \"auth-test-model-82db1a-kserve-565895787-fll8m\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:37.001821 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.001648 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-model-cache\") pod \"auth-test-model-82db1a-kserve-565895787-fll8m\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:37.001821 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.001757 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46a86dc7-e62d-4499-88a5-6a29cbb1b583-tls-certs\") pod \"auth-test-model-82db1a-kserve-565895787-fll8m\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:37.001821 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.001800 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-home\") pod \"auth-test-model-82db1a-kserve-565895787-fll8m\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:37.001958 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.001839 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-dshm\") pod \"auth-test-model-82db1a-kserve-565895787-fll8m\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:37.001958 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.001854 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7npmm\" (UniqueName: \"kubernetes.io/projected/46a86dc7-e62d-4499-88a5-6a29cbb1b583-kube-api-access-7npmm\") pod \"auth-test-model-82db1a-kserve-565895787-fll8m\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:37.103393 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.103339 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46a86dc7-e62d-4499-88a5-6a29cbb1b583-tls-certs\") pod \"auth-test-model-82db1a-kserve-565895787-fll8m\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:37.103393 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.103398 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-home\") pod \"auth-test-model-82db1a-kserve-565895787-fll8m\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:37.103646 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.103424 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-dshm\") pod \"auth-test-model-82db1a-kserve-565895787-fll8m\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:37.103646 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.103441 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7npmm\" (UniqueName: \"kubernetes.io/projected/46a86dc7-e62d-4499-88a5-6a29cbb1b583-kube-api-access-7npmm\") pod \"auth-test-model-82db1a-kserve-565895787-fll8m\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:37.103646 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.103475 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-tmp-dir\") pod \"auth-test-model-82db1a-kserve-565895787-fll8m\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:37.103646 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.103513 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-model-cache\") pod \"auth-test-model-82db1a-kserve-565895787-fll8m\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:37.103888 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.103850 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-home\") pod \"auth-test-model-82db1a-kserve-565895787-fll8m\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:37.103955 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.103875 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-model-cache\") pod \"auth-test-model-82db1a-kserve-565895787-fll8m\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:37.104009 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.103994 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-tmp-dir\") pod \"auth-test-model-82db1a-kserve-565895787-fll8m\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:37.105740 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.105714 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-dshm\") pod \"auth-test-model-82db1a-kserve-565895787-fll8m\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:37.106437 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.106344 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46a86dc7-e62d-4499-88a5-6a29cbb1b583-tls-certs\") pod \"auth-test-model-82db1a-kserve-565895787-fll8m\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:37.115933 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.115906 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7npmm\" (UniqueName: \"kubernetes.io/projected/46a86dc7-e62d-4499-88a5-6a29cbb1b583-kube-api-access-7npmm\") pod \"auth-test-model-82db1a-kserve-565895787-fll8m\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:37.168993 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.168952 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:37.314217 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.314149 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m"] Jul 07 14:59:37.316808 ip-10-0-128-34 kubenswrapper[2562]: W0707 14:59:37.316778 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46a86dc7_e62d_4499_88a5_6a29cbb1b583.slice/crio-895e74ffefb24e2feb25d1cce2abd34234c23942ba9d9f421341726860aaf48a WatchSource:0}: Error finding container 895e74ffefb24e2feb25d1cce2abd34234c23942ba9d9f421341726860aaf48a: Status 404 returned error can't find the container with id 895e74ffefb24e2feb25d1cce2abd34234c23942ba9d9f421341726860aaf48a Jul 07 14:59:37.775672 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.775632 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" event={"ID":"46a86dc7-e62d-4499-88a5-6a29cbb1b583","Type":"ContainerStarted","Data":"685594bc3955d647585bf7489e1268b5ea018fe37df40ebf2c196048f6d7206a"} Jul 07 14:59:37.775672 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.775678 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" event={"ID":"46a86dc7-e62d-4499-88a5-6a29cbb1b583","Type":"ContainerStarted","Data":"895e74ffefb24e2feb25d1cce2abd34234c23942ba9d9f421341726860aaf48a"} Jul 07 14:59:37.795828 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:37.795774 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" podStartSLOduration=1.7957582479999998 podStartE2EDuration="1.795758248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 14:59:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 14:59:37.794557654 +0000 UTC m=+2101.604105143" watchObservedRunningTime="2026-07-07 14:59:37.795758248 +0000 UTC m=+2101.605305736" Jul 07 14:59:38.730107 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:38.730066 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:38.799396 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:38.799336 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:45.123784 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:45.123747 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:45.219604 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:45.219560 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 14:59:47.169844 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:47.169809 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:47.170265 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:47.169959 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:47.182687 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:47.182662 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 14:59:47.834646 ip-10-0-128-34 kubenswrapper[2562]: I0707 14:59:47.834617 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 15:00:11.370842 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.370802 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz"] Jul 07 15:00:11.374151 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.374134 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.376704 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.376675 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-537d76\"/\"kube-root-ca.crt\"" Jul 07 15:00:11.377712 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.377683 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-537d76\"/\"openshift-service-ca.crt\"" Jul 07 15:00:11.377712 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.377705 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-537d76\"/\"auth-test-model-537d76-kserve-self-signed-certs\"" Jul 07 15:00:11.377906 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.377743 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-537d76\"/\"default-dockercfg-pswrt\"" Jul 07 15:00:11.385249 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.385224 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz"] Jul 07 15:00:11.404765 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.404726 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:00:11.432874 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.432839 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-home\") pod \"auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.433064 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.432886 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-model-cache\") pod \"auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.433064 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.433006 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-tmp-dir\") pod \"auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.433064 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.433051 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tst25\" (UniqueName: \"kubernetes.io/projected/89f12f6d-94c6-49e5-ae3a-6580db5583a9-kube-api-access-tst25\") pod \"auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.433233 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.433205 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f12f6d-94c6-49e5-ae3a-6580db5583a9-tls-certs\") pod \"auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.433318 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.433286 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-dshm\") pod \"auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.534918 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.534873 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-dshm\") pod \"auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.535125 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.534930 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-home\") pod \"auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.535125 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.534965 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-model-cache\") pod \"auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.535125 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.534999 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-tmp-dir\") pod \"auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.535125 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.535031 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tst25\" (UniqueName: \"kubernetes.io/projected/89f12f6d-94c6-49e5-ae3a-6580db5583a9-kube-api-access-tst25\") pod \"auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.535335 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.535211 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f12f6d-94c6-49e5-ae3a-6580db5583a9-tls-certs\") pod \"auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.541398 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.535978 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-tmp-dir\") pod \"auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.541398 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.535996 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-model-cache\") pod \"auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.541398 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.536062 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-home\") pod \"auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.541398 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.538672 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f12f6d-94c6-49e5-ae3a-6580db5583a9-tls-certs\") pod \"auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.542900 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.542870 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-dshm\") pod \"auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.545458 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.545432 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tst25\" (UniqueName: \"kubernetes.io/projected/89f12f6d-94c6-49e5-ae3a-6580db5583a9-kube-api-access-tst25\") pod \"auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.684572 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.684534 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:11.826764 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.826663 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz"] Jul 07 15:00:11.829726 ip-10-0-128-34 kubenswrapper[2562]: W0707 15:00:11.829687 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89f12f6d_94c6_49e5_ae3a_6580db5583a9.slice/crio-70513479b97209b53a4a371817f6d960e19cde76d51cc372d4761d6702c7f4a9 WatchSource:0}: Error finding container 70513479b97209b53a4a371817f6d960e19cde76d51cc372d4761d6702c7f4a9: Status 404 returned error can't find the container with id 70513479b97209b53a4a371817f6d960e19cde76d51cc372d4761d6702c7f4a9 Jul 07 15:00:11.944901 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.944798 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" event={"ID":"89f12f6d-94c6-49e5-ae3a-6580db5583a9","Type":"ContainerStarted","Data":"e4569e780d2bbfe767b92d3d93fde61af331fc851da6809f2e9c7b34f707e847"} Jul 07 15:00:11.944901 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.944840 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" event={"ID":"89f12f6d-94c6-49e5-ae3a-6580db5583a9","Type":"ContainerStarted","Data":"70513479b97209b53a4a371817f6d960e19cde76d51cc372d4761d6702c7f4a9"} Jul 07 15:00:11.966806 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:11.966732 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" podStartSLOduration=0.966711004 podStartE2EDuration="966.711004ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:00:11.964931027 +0000 UTC m=+2135.774478514" watchObservedRunningTime="2026-07-07 15:00:11.966711004 +0000 UTC m=+2135.776258493" Jul 07 15:00:21.685643 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:21.685595 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:21.686109 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:21.685661 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:21.698292 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:21.698264 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:22.004765 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:22.004678 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:00:55.312498 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:00:55.312457 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:02:12.955993 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:12.955962 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m"] Jul 07 15:02:13.887058 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:13.887023 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz"] Jul 07 15:02:13.947279 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:13.947249 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:02:14.024482 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:14.024443 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:02:22.935029 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:22.934990 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:02:27.975982 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:27.975937 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" podUID="46a86dc7-e62d-4499-88a5-6a29cbb1b583" containerName="main" containerID="cri-o://685594bc3955d647585bf7489e1268b5ea018fe37df40ebf2c196048f6d7206a" gracePeriod=45 Jul 07 15:02:28.225078 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.225049 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 15:02:28.276450 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.276343 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-dshm\") pod \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " Jul 07 15:02:28.276450 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.276425 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46a86dc7-e62d-4499-88a5-6a29cbb1b583-tls-certs\") pod \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " Jul 07 15:02:28.276671 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.276540 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7npmm\" (UniqueName: \"kubernetes.io/projected/46a86dc7-e62d-4499-88a5-6a29cbb1b583-kube-api-access-7npmm\") pod \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " Jul 07 15:02:28.276671 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.276580 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-model-cache\") pod \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " Jul 07 15:02:28.276671 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.276627 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-home\") pod \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " Jul 07 15:02:28.276843 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.276671 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-tmp-dir\") pod \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\" (UID: \"46a86dc7-e62d-4499-88a5-6a29cbb1b583\") " Jul 07 15:02:28.276936 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.276909 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-model-cache" (OuterVolumeSpecName: "model-cache") pod "46a86dc7-e62d-4499-88a5-6a29cbb1b583" (UID: "46a86dc7-e62d-4499-88a5-6a29cbb1b583"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:02:28.277005 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.276945 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-home" (OuterVolumeSpecName: "home") pod "46a86dc7-e62d-4499-88a5-6a29cbb1b583" (UID: "46a86dc7-e62d-4499-88a5-6a29cbb1b583"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:02:28.277106 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.277088 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-model-cache\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:02:28.277166 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.277112 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-home\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:02:28.277166 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.277118 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "46a86dc7-e62d-4499-88a5-6a29cbb1b583" (UID: "46a86dc7-e62d-4499-88a5-6a29cbb1b583"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:02:28.278734 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.278709 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46a86dc7-e62d-4499-88a5-6a29cbb1b583-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "46a86dc7-e62d-4499-88a5-6a29cbb1b583" (UID: "46a86dc7-e62d-4499-88a5-6a29cbb1b583"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:02:28.278860 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.278754 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a86dc7-e62d-4499-88a5-6a29cbb1b583-kube-api-access-7npmm" (OuterVolumeSpecName: "kube-api-access-7npmm") pod "46a86dc7-e62d-4499-88a5-6a29cbb1b583" (UID: "46a86dc7-e62d-4499-88a5-6a29cbb1b583"). InnerVolumeSpecName "kube-api-access-7npmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:02:28.278860 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.278793 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-dshm" (OuterVolumeSpecName: "dshm") pod "46a86dc7-e62d-4499-88a5-6a29cbb1b583" (UID: "46a86dc7-e62d-4499-88a5-6a29cbb1b583"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:02:28.377825 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.377792 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7npmm\" (UniqueName: \"kubernetes.io/projected/46a86dc7-e62d-4499-88a5-6a29cbb1b583-kube-api-access-7npmm\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:02:28.377825 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.377821 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-tmp-dir\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:02:28.377825 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.377832 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46a86dc7-e62d-4499-88a5-6a29cbb1b583-dshm\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:02:28.378057 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.377841 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46a86dc7-e62d-4499-88a5-6a29cbb1b583-tls-certs\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:02:28.621301 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.621209 2562 generic.go:356] "Generic (PLEG): container finished" podID="46a86dc7-e62d-4499-88a5-6a29cbb1b583" containerID="685594bc3955d647585bf7489e1268b5ea018fe37df40ebf2c196048f6d7206a" exitCode=0 Jul 07 15:02:28.621301 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.621253 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" Jul 07 15:02:28.621301 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.621279 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" event={"ID":"46a86dc7-e62d-4499-88a5-6a29cbb1b583","Type":"ContainerDied","Data":"685594bc3955d647585bf7489e1268b5ea018fe37df40ebf2c196048f6d7206a"} Jul 07 15:02:28.621619 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.621319 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m" event={"ID":"46a86dc7-e62d-4499-88a5-6a29cbb1b583","Type":"ContainerDied","Data":"895e74ffefb24e2feb25d1cce2abd34234c23942ba9d9f421341726860aaf48a"} Jul 07 15:02:28.621619 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.621340 2562 scope.go:117] "RemoveContainer" containerID="685594bc3955d647585bf7489e1268b5ea018fe37df40ebf2c196048f6d7206a" Jul 07 15:02:28.630928 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.630909 2562 scope.go:117] "RemoveContainer" containerID="685594bc3955d647585bf7489e1268b5ea018fe37df40ebf2c196048f6d7206a" Jul 07 15:02:28.631185 ip-10-0-128-34 kubenswrapper[2562]: E0707 15:02:28.631166 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"685594bc3955d647585bf7489e1268b5ea018fe37df40ebf2c196048f6d7206a\": container with ID starting with 685594bc3955d647585bf7489e1268b5ea018fe37df40ebf2c196048f6d7206a not found: ID does not exist" containerID="685594bc3955d647585bf7489e1268b5ea018fe37df40ebf2c196048f6d7206a" Jul 07 15:02:28.631224 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.631195 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"685594bc3955d647585bf7489e1268b5ea018fe37df40ebf2c196048f6d7206a"} err="failed to get container status \"685594bc3955d647585bf7489e1268b5ea018fe37df40ebf2c196048f6d7206a\": rpc error: code = NotFound desc = could not find container \"685594bc3955d647585bf7489e1268b5ea018fe37df40ebf2c196048f6d7206a\": container with ID starting with 685594bc3955d647585bf7489e1268b5ea018fe37df40ebf2c196048f6d7206a not found: ID does not exist" Jul 07 15:02:28.645256 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.645236 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m"] Jul 07 15:02:28.652430 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.652407 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-82db1a/auth-test-model-82db1a-kserve-565895787-fll8m"] Jul 07 15:02:28.707493 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.707461 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46a86dc7-e62d-4499-88a5-6a29cbb1b583" path="/var/lib/kubelet/pods/46a86dc7-e62d-4499-88a5-6a29cbb1b583/volumes" Jul 07 15:02:28.906758 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:28.906703 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" podUID="89f12f6d-94c6-49e5-ae3a-6580db5583a9" containerName="main" containerID="cri-o://e4569e780d2bbfe767b92d3d93fde61af331fc851da6809f2e9c7b34f707e847" gracePeriod=45 Jul 07 15:02:29.163768 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.163713 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:02:29.286997 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.286965 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-model-cache\") pod \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " Jul 07 15:02:29.287180 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.287030 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tst25\" (UniqueName: \"kubernetes.io/projected/89f12f6d-94c6-49e5-ae3a-6580db5583a9-kube-api-access-tst25\") pod \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " Jul 07 15:02:29.287180 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.287088 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f12f6d-94c6-49e5-ae3a-6580db5583a9-tls-certs\") pod \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " Jul 07 15:02:29.287180 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.287126 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-tmp-dir\") pod \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " Jul 07 15:02:29.287180 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.287153 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-home\") pod \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " Jul 07 15:02:29.287180 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.287179 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-dshm\") pod \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\" (UID: \"89f12f6d-94c6-49e5-ae3a-6580db5583a9\") " Jul 07 15:02:29.287494 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.287256 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "89f12f6d-94c6-49e5-ae3a-6580db5583a9" (UID: "89f12f6d-94c6-49e5-ae3a-6580db5583a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:02:29.287494 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.287395 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "89f12f6d-94c6-49e5-ae3a-6580db5583a9" (UID: "89f12f6d-94c6-49e5-ae3a-6580db5583a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:02:29.287494 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.287446 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-home" (OuterVolumeSpecName: "home") pod "89f12f6d-94c6-49e5-ae3a-6580db5583a9" (UID: "89f12f6d-94c6-49e5-ae3a-6580db5583a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:02:29.287613 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.287584 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-model-cache\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:02:29.287613 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.287605 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-tmp-dir\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:02:29.287705 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.287619 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-home\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:02:29.289345 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.289321 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-dshm" (OuterVolumeSpecName: "dshm") pod "89f12f6d-94c6-49e5-ae3a-6580db5583a9" (UID: "89f12f6d-94c6-49e5-ae3a-6580db5583a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:02:29.289457 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.289344 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f12f6d-94c6-49e5-ae3a-6580db5583a9-kube-api-access-tst25" (OuterVolumeSpecName: "kube-api-access-tst25") pod "89f12f6d-94c6-49e5-ae3a-6580db5583a9" (UID: "89f12f6d-94c6-49e5-ae3a-6580db5583a9"). InnerVolumeSpecName "kube-api-access-tst25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:02:29.289457 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.289360 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89f12f6d-94c6-49e5-ae3a-6580db5583a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "89f12f6d-94c6-49e5-ae3a-6580db5583a9" (UID: "89f12f6d-94c6-49e5-ae3a-6580db5583a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:02:29.388859 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.388825 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tst25\" (UniqueName: \"kubernetes.io/projected/89f12f6d-94c6-49e5-ae3a-6580db5583a9-kube-api-access-tst25\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:02:29.388859 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.388858 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89f12f6d-94c6-49e5-ae3a-6580db5583a9-tls-certs\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:02:29.388859 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.388868 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89f12f6d-94c6-49e5-ae3a-6580db5583a9-dshm\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:02:29.627696 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.627607 2562 generic.go:356] "Generic (PLEG): container finished" podID="89f12f6d-94c6-49e5-ae3a-6580db5583a9" containerID="e4569e780d2bbfe767b92d3d93fde61af331fc851da6809f2e9c7b34f707e847" exitCode=0 Jul 07 15:02:29.627696 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.627679 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" Jul 07 15:02:29.627933 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.627689 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" event={"ID":"89f12f6d-94c6-49e5-ae3a-6580db5583a9","Type":"ContainerDied","Data":"e4569e780d2bbfe767b92d3d93fde61af331fc851da6809f2e9c7b34f707e847"} Jul 07 15:02:29.627933 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.627732 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz" event={"ID":"89f12f6d-94c6-49e5-ae3a-6580db5583a9","Type":"ContainerDied","Data":"70513479b97209b53a4a371817f6d960e19cde76d51cc372d4761d6702c7f4a9"} Jul 07 15:02:29.627933 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.627755 2562 scope.go:117] "RemoveContainer" containerID="e4569e780d2bbfe767b92d3d93fde61af331fc851da6809f2e9c7b34f707e847" Jul 07 15:02:29.637609 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.637588 2562 scope.go:117] "RemoveContainer" containerID="e4569e780d2bbfe767b92d3d93fde61af331fc851da6809f2e9c7b34f707e847" Jul 07 15:02:29.637875 ip-10-0-128-34 kubenswrapper[2562]: E0707 15:02:29.637858 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4569e780d2bbfe767b92d3d93fde61af331fc851da6809f2e9c7b34f707e847\": container with ID starting with e4569e780d2bbfe767b92d3d93fde61af331fc851da6809f2e9c7b34f707e847 not found: ID does not exist" containerID="e4569e780d2bbfe767b92d3d93fde61af331fc851da6809f2e9c7b34f707e847" Jul 07 15:02:29.637946 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.637880 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4569e780d2bbfe767b92d3d93fde61af331fc851da6809f2e9c7b34f707e847"} err="failed to get container status \"e4569e780d2bbfe767b92d3d93fde61af331fc851da6809f2e9c7b34f707e847\": rpc error: code = NotFound desc = could not find container \"e4569e780d2bbfe767b92d3d93fde61af331fc851da6809f2e9c7b34f707e847\": container with ID starting with e4569e780d2bbfe767b92d3d93fde61af331fc851da6809f2e9c7b34f707e847 not found: ID does not exist" Jul 07 15:02:29.656873 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.656848 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz"] Jul 07 15:02:29.663850 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:29.663824 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-537d76/auth-test-model-537d76-kserve-5cdc7b9ddc-kwtzz"] Jul 07 15:02:30.707646 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:02:30.707614 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89f12f6d-94c6-49e5-ae3a-6580db5583a9" path="/var/lib/kubelet/pods/89f12f6d-94c6-49e5-ae3a-6580db5583a9/volumes" Jul 07 15:03:07.381000 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:07.380962 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:03:11.952737 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:11.952699 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:03:12.053515 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:12.053476 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:03:15.288802 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:15.288769 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:03:25.682853 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.682820 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4"] Jul 07 15:03:25.683263 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.683215 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46a86dc7-e62d-4499-88a5-6a29cbb1b583" containerName="main" Jul 07 15:03:25.683263 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.683229 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="46a86dc7-e62d-4499-88a5-6a29cbb1b583" containerName="main" Jul 07 15:03:25.683263 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.683241 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f12f6d-94c6-49e5-ae3a-6580db5583a9" containerName="main" Jul 07 15:03:25.683263 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.683247 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f12f6d-94c6-49e5-ae3a-6580db5583a9" containerName="main" Jul 07 15:03:25.683417 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.683308 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="89f12f6d-94c6-49e5-ae3a-6580db5583a9" containerName="main" Jul 07 15:03:25.683417 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.683316 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="46a86dc7-e62d-4499-88a5-6a29cbb1b583" containerName="main" Jul 07 15:03:25.686083 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.686069 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:25.690035 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.690011 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-537d76\"/\"kube-root-ca.crt\"" Jul 07 15:03:25.690035 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.690025 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-537d76\"/\"openshift-service-ca.crt\"" Jul 07 15:03:25.690035 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.690031 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-537d76\"/\"default-dockercfg-pswrt\"" Jul 07 15:03:25.690237 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.690028 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-537d76\"/\"sub-test-model-537d76-kserve-self-signed-certs\"" Jul 07 15:03:25.702285 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.702263 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4"] Jul 07 15:03:25.731004 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.730964 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:03:25.834193 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.834163 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-model-cache\") pod \"sub-test-model-537d76-kserve-b6dffffd7-zkmj4\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:25.834402 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.834208 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-tmp-dir\") pod \"sub-test-model-537d76-kserve-b6dffffd7-zkmj4\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:25.834402 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.834327 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-dshm\") pod \"sub-test-model-537d76-kserve-b6dffffd7-zkmj4\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:25.834402 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.834368 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5393d83-054f-4195-a8d8-c6b0b2acb70d-tls-certs\") pod \"sub-test-model-537d76-kserve-b6dffffd7-zkmj4\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:25.834554 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.834416 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhcv2\" (UniqueName: \"kubernetes.io/projected/b5393d83-054f-4195-a8d8-c6b0b2acb70d-kube-api-access-mhcv2\") pod \"sub-test-model-537d76-kserve-b6dffffd7-zkmj4\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:25.834554 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.834459 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-home\") pod \"sub-test-model-537d76-kserve-b6dffffd7-zkmj4\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:25.935619 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.935523 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-home\") pod \"sub-test-model-537d76-kserve-b6dffffd7-zkmj4\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:25.935619 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.935569 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-model-cache\") pod \"sub-test-model-537d76-kserve-b6dffffd7-zkmj4\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:25.935619 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.935617 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-tmp-dir\") pod \"sub-test-model-537d76-kserve-b6dffffd7-zkmj4\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:25.935896 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.935694 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-dshm\") pod \"sub-test-model-537d76-kserve-b6dffffd7-zkmj4\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:25.935896 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.935734 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5393d83-054f-4195-a8d8-c6b0b2acb70d-tls-certs\") pod \"sub-test-model-537d76-kserve-b6dffffd7-zkmj4\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:25.935896 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.935760 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhcv2\" (UniqueName: \"kubernetes.io/projected/b5393d83-054f-4195-a8d8-c6b0b2acb70d-kube-api-access-mhcv2\") pod \"sub-test-model-537d76-kserve-b6dffffd7-zkmj4\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:25.936048 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.935940 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-home\") pod \"sub-test-model-537d76-kserve-b6dffffd7-zkmj4\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:25.936048 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.935965 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-model-cache\") pod \"sub-test-model-537d76-kserve-b6dffffd7-zkmj4\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:25.936048 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.936022 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-tmp-dir\") pod \"sub-test-model-537d76-kserve-b6dffffd7-zkmj4\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:25.937961 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.937932 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-dshm\") pod \"sub-test-model-537d76-kserve-b6dffffd7-zkmj4\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:25.938118 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.938102 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5393d83-054f-4195-a8d8-c6b0b2acb70d-tls-certs\") pod \"sub-test-model-537d76-kserve-b6dffffd7-zkmj4\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:25.944564 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.944544 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhcv2\" (UniqueName: \"kubernetes.io/projected/b5393d83-054f-4195-a8d8-c6b0b2acb70d-kube-api-access-mhcv2\") pod \"sub-test-model-537d76-kserve-b6dffffd7-zkmj4\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:25.995535 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:25.995494 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:26.136350 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:26.136324 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4"] Jul 07 15:03:26.138595 ip-10-0-128-34 kubenswrapper[2562]: W0707 15:03:26.138570 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5393d83_054f_4195_a8d8_c6b0b2acb70d.slice/crio-1c5504350cd78a8dfd91bcd5f24ab17f7d1b5fdac7927230146f9839430b663d WatchSource:0}: Error finding container 1c5504350cd78a8dfd91bcd5f24ab17f7d1b5fdac7927230146f9839430b663d: Status 404 returned error can't find the container with id 1c5504350cd78a8dfd91bcd5f24ab17f7d1b5fdac7927230146f9839430b663d Jul 07 15:03:26.902974 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:26.902937 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" event={"ID":"b5393d83-054f-4195-a8d8-c6b0b2acb70d","Type":"ContainerStarted","Data":"de717632032cf011b4d6d33198988ab83511150be468fec4d270081c095ba43a"} Jul 07 15:03:26.903414 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:26.902980 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" event={"ID":"b5393d83-054f-4195-a8d8-c6b0b2acb70d","Type":"ContainerStarted","Data":"1c5504350cd78a8dfd91bcd5f24ab17f7d1b5fdac7927230146f9839430b663d"} Jul 07 15:03:26.941146 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:26.941094 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" podStartSLOduration=1.941077138 podStartE2EDuration="1.941077138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:03:26.939888613 +0000 UTC m=+2330.749436104" watchObservedRunningTime="2026-07-07 15:03:26.941077138 +0000 UTC m=+2330.750624627" Jul 07 15:03:35.995902 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:35.995808 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:35.995902 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:35.995879 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:36.008518 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:36.008494 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:36.961964 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:36.961936 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:03:58.989532 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:58.989490 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:03:59.880264 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:59.880224 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4"] Jul 07 15:03:59.907265 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:03:59.907219 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:12.172636 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.172598 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4"] Jul 07 15:04:12.176028 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.176001 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.179346 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.179319 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-82db1a\"/\"default-dockercfg-x5scf\"" Jul 07 15:04:12.179504 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.179349 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-82db1a\"/\"kube-root-ca.crt\"" Jul 07 15:04:12.179504 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.179438 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-82db1a\"/\"openshift-service-ca.crt\"" Jul 07 15:04:12.179504 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.179441 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-82db1a\"/\"rate-test-model-82db1a-kserve-self-signed-certs\"" Jul 07 15:04:12.186411 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.186389 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4"] Jul 07 15:04:12.213471 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.213442 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:12.267707 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.267678 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-model-cache\") pod \"rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.267875 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.267733 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5rk9\" (UniqueName: \"kubernetes.io/projected/c5ef9286-65f6-4342-87cc-aa29c0c01eac-kube-api-access-d5rk9\") pod \"rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.267875 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.267836 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-dshm\") pod \"rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.267996 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.267891 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-home\") pod \"rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.267996 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.267948 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ef9286-65f6-4342-87cc-aa29c0c01eac-tls-certs\") pod \"rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.268075 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.267997 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-tmp-dir\") pod \"rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.368929 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.368899 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-dshm\") pod \"rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.369128 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.368950 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-home\") pod \"rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.369128 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.368998 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ef9286-65f6-4342-87cc-aa29c0c01eac-tls-certs\") pod \"rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.369128 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.369044 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-tmp-dir\") pod \"rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.369128 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.369073 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-model-cache\") pod \"rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.369340 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.369136 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5rk9\" (UniqueName: \"kubernetes.io/projected/c5ef9286-65f6-4342-87cc-aa29c0c01eac-kube-api-access-d5rk9\") pod \"rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.369446 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.369418 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-home\") pod \"rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.369519 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.369479 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-tmp-dir\") pod \"rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.369519 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.369497 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-model-cache\") pod \"rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.371310 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.371288 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-dshm\") pod \"rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.371441 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.371426 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ef9286-65f6-4342-87cc-aa29c0c01eac-tls-certs\") pod \"rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.379501 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.379478 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5rk9\" (UniqueName: \"kubernetes.io/projected/c5ef9286-65f6-4342-87cc-aa29c0c01eac-kube-api-access-d5rk9\") pod \"rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.486931 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.486845 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:12.625055 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:12.625027 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4"] Jul 07 15:04:12.627617 ip-10-0-128-34 kubenswrapper[2562]: W0707 15:04:12.627589 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5ef9286_65f6_4342_87cc_aa29c0c01eac.slice/crio-6125482b96018712ff828a1bf8b45280895a26d7debb15089bb7e98d22816017 WatchSource:0}: Error finding container 6125482b96018712ff828a1bf8b45280895a26d7debb15089bb7e98d22816017: Status 404 returned error can't find the container with id 6125482b96018712ff828a1bf8b45280895a26d7debb15089bb7e98d22816017 Jul 07 15:04:13.123531 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:13.123494 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" event={"ID":"c5ef9286-65f6-4342-87cc-aa29c0c01eac","Type":"ContainerStarted","Data":"794addaed73d3f50cb1d516510806765e02c2e000d14ae009fd721d7a3766601"} Jul 07 15:04:13.123707 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:13.123537 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" event={"ID":"c5ef9286-65f6-4342-87cc-aa29c0c01eac","Type":"ContainerStarted","Data":"6125482b96018712ff828a1bf8b45280895a26d7debb15089bb7e98d22816017"} Jul 07 15:04:13.146364 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:13.146312 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" podStartSLOduration=1.146297046 podStartE2EDuration="1.146297046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 15:04:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:04:13.145457053 +0000 UTC m=+2376.955004563" watchObservedRunningTime="2026-07-07 15:04:13.146297046 +0000 UTC m=+2376.955844536" Jul 07 15:04:14.900276 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:14.900228 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" podUID="b5393d83-054f-4195-a8d8-c6b0b2acb70d" containerName="main" containerID="cri-o://de717632032cf011b4d6d33198988ab83511150be468fec4d270081c095ba43a" gracePeriod=45 Jul 07 15:04:15.135444 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.135415 2562 generic.go:356] "Generic (PLEG): container finished" podID="b5393d83-054f-4195-a8d8-c6b0b2acb70d" containerID="de717632032cf011b4d6d33198988ab83511150be468fec4d270081c095ba43a" exitCode=0 Jul 07 15:04:15.135594 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.135493 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" event={"ID":"b5393d83-054f-4195-a8d8-c6b0b2acb70d","Type":"ContainerDied","Data":"de717632032cf011b4d6d33198988ab83511150be468fec4d270081c095ba43a"} Jul 07 15:04:15.155872 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.155816 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:04:15.197126 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.197091 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-model-cache\") pod \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " Jul 07 15:04:15.197298 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.197190 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-dshm\") pod \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " Jul 07 15:04:15.197399 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.197315 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhcv2\" (UniqueName: \"kubernetes.io/projected/b5393d83-054f-4195-a8d8-c6b0b2acb70d-kube-api-access-mhcv2\") pod \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " Jul 07 15:04:15.197399 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.197344 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-tmp-dir\") pod \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " Jul 07 15:04:15.197522 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.197413 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5393d83-054f-4195-a8d8-c6b0b2acb70d-tls-certs\") pod \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " Jul 07 15:04:15.197522 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.197426 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5393d83-054f-4195-a8d8-c6b0b2acb70d" (UID: "b5393d83-054f-4195-a8d8-c6b0b2acb70d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:04:15.197522 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.197453 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-home\") pod \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\" (UID: \"b5393d83-054f-4195-a8d8-c6b0b2acb70d\") " Jul 07 15:04:15.197700 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.197660 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5393d83-054f-4195-a8d8-c6b0b2acb70d" (UID: "b5393d83-054f-4195-a8d8-c6b0b2acb70d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:04:15.197775 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.197728 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-home" (OuterVolumeSpecName: "home") pod "b5393d83-054f-4195-a8d8-c6b0b2acb70d" (UID: "b5393d83-054f-4195-a8d8-c6b0b2acb70d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:04:15.197842 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.197827 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-tmp-dir\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:04:15.197889 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.197847 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-home\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:04:15.197889 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.197861 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-model-cache\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:04:15.199508 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.199484 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5393d83-054f-4195-a8d8-c6b0b2acb70d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5393d83-054f-4195-a8d8-c6b0b2acb70d" (UID: "b5393d83-054f-4195-a8d8-c6b0b2acb70d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:04:15.199611 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.199599 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5393d83-054f-4195-a8d8-c6b0b2acb70d-kube-api-access-mhcv2" (OuterVolumeSpecName: "kube-api-access-mhcv2") pod "b5393d83-054f-4195-a8d8-c6b0b2acb70d" (UID: "b5393d83-054f-4195-a8d8-c6b0b2acb70d"). InnerVolumeSpecName "kube-api-access-mhcv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:04:15.199755 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.199740 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-dshm" (OuterVolumeSpecName: "dshm") pod "b5393d83-054f-4195-a8d8-c6b0b2acb70d" (UID: "b5393d83-054f-4195-a8d8-c6b0b2acb70d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:04:15.299278 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.299246 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5393d83-054f-4195-a8d8-c6b0b2acb70d-dshm\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:04:15.299278 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.299275 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhcv2\" (UniqueName: \"kubernetes.io/projected/b5393d83-054f-4195-a8d8-c6b0b2acb70d-kube-api-access-mhcv2\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:04:15.299278 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:15.299284 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5393d83-054f-4195-a8d8-c6b0b2acb70d-tls-certs\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:04:16.141209 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:16.141174 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" event={"ID":"b5393d83-054f-4195-a8d8-c6b0b2acb70d","Type":"ContainerDied","Data":"1c5504350cd78a8dfd91bcd5f24ab17f7d1b5fdac7927230146f9839430b663d"} Jul 07 15:04:16.141615 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:16.141183 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4" Jul 07 15:04:16.141615 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:16.141222 2562 scope.go:117] "RemoveContainer" containerID="de717632032cf011b4d6d33198988ab83511150be468fec4d270081c095ba43a" Jul 07 15:04:16.167310 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:16.167279 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4"] Jul 07 15:04:16.174760 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:16.174733 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-537d76/sub-test-model-537d76-kserve-b6dffffd7-zkmj4"] Jul 07 15:04:16.712497 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:16.712465 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5393d83-054f-4195-a8d8-c6b0b2acb70d" path="/var/lib/kubelet/pods/b5393d83-054f-4195-a8d8-c6b0b2acb70d/volumes" Jul 07 15:04:22.487384 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:22.487340 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:22.487803 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:22.487416 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:22.500214 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:22.500187 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:23.189953 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:23.189862 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:04:27.554057 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:27.554019 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:31.705460 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:31.705424 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:39.573344 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:39.573307 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:39.673529 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:39.673492 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:45.866984 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:45.866916 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:04:45.966777 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:04:45.966745 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:21.701719 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:21.701685 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4"] Jul 07 15:05:21.798056 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:21.798012 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:22.709451 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:22.709406 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:23.744262 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:23.744226 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:24.259525 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:24.259480 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:24.358704 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:24.358667 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:24.462397 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:24.462337 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:24.588582 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:24.588480 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:29.206830 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:29.206789 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:05:36.721137 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:36.721096 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" podUID="c5ef9286-65f6-4342-87cc-aa29c0c01eac" containerName="main" containerID="cri-o://794addaed73d3f50cb1d516510806765e02c2e000d14ae009fd721d7a3766601" gracePeriod=45 Jul 07 15:05:36.972958 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:36.972891 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:05:37.100553 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.100517 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5rk9\" (UniqueName: \"kubernetes.io/projected/c5ef9286-65f6-4342-87cc-aa29c0c01eac-kube-api-access-d5rk9\") pod \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " Jul 07 15:05:37.100743 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.100578 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-tmp-dir\") pod \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " Jul 07 15:05:37.100743 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.100615 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-dshm\") pod \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " Jul 07 15:05:37.100743 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.100666 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ef9286-65f6-4342-87cc-aa29c0c01eac-tls-certs\") pod \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " Jul 07 15:05:37.100743 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.100712 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-model-cache\") pod \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " Jul 07 15:05:37.100960 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.100762 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-home\") pod \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\" (UID: \"c5ef9286-65f6-4342-87cc-aa29c0c01eac\") " Jul 07 15:05:37.100960 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.100925 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c5ef9286-65f6-4342-87cc-aa29c0c01eac" (UID: "c5ef9286-65f6-4342-87cc-aa29c0c01eac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:05:37.101074 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.100977 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-model-cache" (OuterVolumeSpecName: "model-cache") pod "c5ef9286-65f6-4342-87cc-aa29c0c01eac" (UID: "c5ef9286-65f6-4342-87cc-aa29c0c01eac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:05:37.101142 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.101119 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-home" (OuterVolumeSpecName: "home") pod "c5ef9286-65f6-4342-87cc-aa29c0c01eac" (UID: "c5ef9286-65f6-4342-87cc-aa29c0c01eac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:05:37.101217 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.101194 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-model-cache\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:05:37.101217 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.101217 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-home\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:05:37.101420 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.101232 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-tmp-dir\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:05:37.102617 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.102590 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ef9286-65f6-4342-87cc-aa29c0c01eac-kube-api-access-d5rk9" (OuterVolumeSpecName: "kube-api-access-d5rk9") pod "c5ef9286-65f6-4342-87cc-aa29c0c01eac" (UID: "c5ef9286-65f6-4342-87cc-aa29c0c01eac"). InnerVolumeSpecName "kube-api-access-d5rk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:05:37.102742 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.102725 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ef9286-65f6-4342-87cc-aa29c0c01eac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c5ef9286-65f6-4342-87cc-aa29c0c01eac" (UID: "c5ef9286-65f6-4342-87cc-aa29c0c01eac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:05:37.102809 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.102788 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-dshm" (OuterVolumeSpecName: "dshm") pod "c5ef9286-65f6-4342-87cc-aa29c0c01eac" (UID: "c5ef9286-65f6-4342-87cc-aa29c0c01eac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 07 15:05:37.202231 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.202192 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5rk9\" (UniqueName: \"kubernetes.io/projected/c5ef9286-65f6-4342-87cc-aa29c0c01eac-kube-api-access-d5rk9\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:05:37.202231 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.202224 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5ef9286-65f6-4342-87cc-aa29c0c01eac-dshm\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:05:37.202231 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.202233 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5ef9286-65f6-4342-87cc-aa29c0c01eac-tls-certs\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:05:37.553531 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.553498 2562 generic.go:356] "Generic (PLEG): container finished" podID="c5ef9286-65f6-4342-87cc-aa29c0c01eac" containerID="794addaed73d3f50cb1d516510806765e02c2e000d14ae009fd721d7a3766601" exitCode=0 Jul 07 15:05:37.553736 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.553550 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" Jul 07 15:05:37.553736 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.553583 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" event={"ID":"c5ef9286-65f6-4342-87cc-aa29c0c01eac","Type":"ContainerDied","Data":"794addaed73d3f50cb1d516510806765e02c2e000d14ae009fd721d7a3766601"} Jul 07 15:05:37.553736 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.553634 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4" event={"ID":"c5ef9286-65f6-4342-87cc-aa29c0c01eac","Type":"ContainerDied","Data":"6125482b96018712ff828a1bf8b45280895a26d7debb15089bb7e98d22816017"} Jul 07 15:05:37.553736 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.553656 2562 scope.go:117] "RemoveContainer" containerID="794addaed73d3f50cb1d516510806765e02c2e000d14ae009fd721d7a3766601" Jul 07 15:05:37.563016 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.562999 2562 scope.go:117] "RemoveContainer" containerID="794addaed73d3f50cb1d516510806765e02c2e000d14ae009fd721d7a3766601" Jul 07 15:05:37.563238 ip-10-0-128-34 kubenswrapper[2562]: E0707 15:05:37.563222 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"794addaed73d3f50cb1d516510806765e02c2e000d14ae009fd721d7a3766601\": container with ID starting with 794addaed73d3f50cb1d516510806765e02c2e000d14ae009fd721d7a3766601 not found: ID does not exist" containerID="794addaed73d3f50cb1d516510806765e02c2e000d14ae009fd721d7a3766601" Jul 07 15:05:37.563287 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.563244 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"794addaed73d3f50cb1d516510806765e02c2e000d14ae009fd721d7a3766601"} err="failed to get container status \"794addaed73d3f50cb1d516510806765e02c2e000d14ae009fd721d7a3766601\": rpc error: code = NotFound desc = could not find container \"794addaed73d3f50cb1d516510806765e02c2e000d14ae009fd721d7a3766601\": container with ID starting with 794addaed73d3f50cb1d516510806765e02c2e000d14ae009fd721d7a3766601 not found: ID does not exist" Jul 07 15:05:37.592096 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.592067 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4"] Jul 07 15:05:37.594394 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:37.594352 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-82db1a/rate-test-model-82db1a-kserve-7bfc6c7c55-f69q4"] Jul 07 15:05:38.706899 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:05:38.706868 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5ef9286-65f6-4342-87cc-aa29c0c01eac" path="/var/lib/kubelet/pods/c5ef9286-65f6-4342-87cc-aa29c0c01eac/volumes" Jul 07 15:06:07.655561 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:07.655523 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:07.946114 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:07.946014 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:08.178242 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:08.178208 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:08.560595 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:08.560563 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:08.978357 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:08.978323 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:09.217553 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:09.217526 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:09.506809 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:09.506777 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472"] Jul 07 15:06:09.507049 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:09.507015 2562 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" podUID="486a1e57-5fd6-446b-b0f8-85fa847b7489" containerName="maas-api" containerID="cri-o://78c8ab0280d25a25561bd5e65b9f05c5e54eb7166b676c79c1a13e1fda21e358" gracePeriod=30 Jul 07 15:06:09.704902 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:09.704875 2562 generic.go:356] "Generic (PLEG): container finished" podID="486a1e57-5fd6-446b-b0f8-85fa847b7489" containerID="78c8ab0280d25a25561bd5e65b9f05c5e54eb7166b676c79c1a13e1fda21e358" exitCode=0 Jul 07 15:06:09.705082 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:09.704949 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" event={"ID":"486a1e57-5fd6-446b-b0f8-85fa847b7489","Type":"ContainerDied","Data":"78c8ab0280d25a25561bd5e65b9f05c5e54eb7166b676c79c1a13e1fda21e358"} Jul 07 15:06:09.757791 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:09.757716 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" Jul 07 15:06:09.905129 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:09.905092 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/486a1e57-5fd6-446b-b0f8-85fa847b7489-openshift-service-ca\") pod \"486a1e57-5fd6-446b-b0f8-85fa847b7489\" (UID: \"486a1e57-5fd6-446b-b0f8-85fa847b7489\") " Jul 07 15:06:09.905311 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:09.905159 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swx8x\" (UniqueName: \"kubernetes.io/projected/486a1e57-5fd6-446b-b0f8-85fa847b7489-kube-api-access-swx8x\") pod \"486a1e57-5fd6-446b-b0f8-85fa847b7489\" (UID: \"486a1e57-5fd6-446b-b0f8-85fa847b7489\") " Jul 07 15:06:09.905311 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:09.905254 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/486a1e57-5fd6-446b-b0f8-85fa847b7489-maas-api-tls\") pod \"486a1e57-5fd6-446b-b0f8-85fa847b7489\" (UID: \"486a1e57-5fd6-446b-b0f8-85fa847b7489\") " Jul 07 15:06:09.905592 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:09.905566 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486a1e57-5fd6-446b-b0f8-85fa847b7489-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "486a1e57-5fd6-446b-b0f8-85fa847b7489" (UID: "486a1e57-5fd6-446b-b0f8-85fa847b7489"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 07 15:06:09.907384 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:09.907353 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486a1e57-5fd6-446b-b0f8-85fa847b7489-kube-api-access-swx8x" (OuterVolumeSpecName: "kube-api-access-swx8x") pod "486a1e57-5fd6-446b-b0f8-85fa847b7489" (UID: "486a1e57-5fd6-446b-b0f8-85fa847b7489"). InnerVolumeSpecName "kube-api-access-swx8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 07 15:06:09.907453 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:09.907354 2562 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/486a1e57-5fd6-446b-b0f8-85fa847b7489-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "486a1e57-5fd6-446b-b0f8-85fa847b7489" (UID: "486a1e57-5fd6-446b-b0f8-85fa847b7489"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 07 15:06:09.915018 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:09.914992 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:09.995428 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:09.995367 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:10.006291 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:10.006270 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/486a1e57-5fd6-446b-b0f8-85fa847b7489-maas-api-tls\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:06:10.006424 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:10.006294 2562 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/486a1e57-5fd6-446b-b0f8-85fa847b7489-openshift-service-ca\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:06:10.006424 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:10.006308 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swx8x\" (UniqueName: \"kubernetes.io/projected/486a1e57-5fd6-446b-b0f8-85fa847b7489-kube-api-access-swx8x\") on node \"ip-10-0-128-34.ec2.internal\" DevicePath \"\"" Jul 07 15:06:10.710261 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:10.710229 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" Jul 07 15:06:10.710261 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:10.710254 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472" event={"ID":"486a1e57-5fd6-446b-b0f8-85fa847b7489","Type":"ContainerDied","Data":"02ccdf561d397c3badb2f5728f54ac86edca2587debb73e658884911de94edf0"} Jul 07 15:06:10.710524 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:10.710307 2562 scope.go:117] "RemoveContainer" containerID="78c8ab0280d25a25561bd5e65b9f05c5e54eb7166b676c79c1a13e1fda21e358" Jul 07 15:06:10.741809 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:10.741782 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472"] Jul 07 15:06:10.748559 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:10.748539 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-537d76-6cf7b5b98d-bf472"] Jul 07 15:06:12.706729 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:12.706693 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="486a1e57-5fd6-446b-b0f8-85fa847b7489" path="/var/lib/kubelet/pods/486a1e57-5fd6-446b-b0f8-85fa847b7489/volumes" Jul 07 15:06:16.869014 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:16.868979 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:16.942934 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:16.942897 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:17.192301 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:17.192270 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8486594fb6-4nw5x_b2e412e5-2397-4f98-8b6a-b8eb08bb77a4/authorino/0.log" Jul 07 15:06:22.301952 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:22.301919 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-64f64495c-wq6t7_3a127701-b325-405f-a500-348b93eefae7/maas-api/0.log" Jul 07 15:06:23.285901 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:23.285824 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f546bc4-m4psb_64f0cd1e-7248-4696-8782-1b544a20848b/copy-manifests/0.log" Jul 07 15:06:23.309931 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:23.309904 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f546bc4-m4psb_64f0cd1e-7248-4696-8782-1b544a20848b/manager/0.log" Jul 07 15:06:23.616614 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:23.616533 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-cvhmk_ecf59abb-a58d-439b-901c-7beab9368363/postgres/0.log" Jul 07 15:06:25.018561 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:25.018528 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8486594fb6-4nw5x_b2e412e5-2397-4f98-8b6a-b8eb08bb77a4/authorino/0.log" Jul 07 15:06:25.422491 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:25.422460 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-6ngls_56a9a81d-3332-4687-ba1c-c32bb4f8e4bd/kuadrant-console-plugin/0.log" Jul 07 15:06:26.758344 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:26.758314 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f67c6995-xtj92_7a2df886-cb1c-43fa-87ef-067cc88830b7/kube-auth-proxy/0.log" Jul 07 15:06:27.038949 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:27.038866 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-5vgm2_f202d8f1-a537-4765-9352-8a229fb5e65f/payload-pre-processing/0.log" Jul 07 15:06:27.791999 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:27.791966 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-6p7w9_075d5305-a170-4da4-a00d-2fbdbb5d5a6c/main/0.log" Jul 07 15:06:28.036002 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:28.035926 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbj9n4w_47d7d674-b461-4be4-ab59-1beff66acbda/main/0.log" Jul 07 15:06:28.296189 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:28.296161 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-q6cv5_495d8e2e-1387-4209-8321-5e3233fb8f52/main/0.log" Jul 07 15:06:32.371622 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:32.371579 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:32.506321 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:32.506282 2562 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" limit=1024 Jul 07 15:06:35.647768 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:35.647738 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7w875_fdf0d65e-e82f-4183-90df-fde1f0ce3435/global-pull-secret-syncer/0.log" Jul 07 15:06:35.886871 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:35.886840 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-88g99_a09e6909-6763-46df-9d1c-1a920a322d59/konnectivity-agent/0.log" Jul 07 15:06:36.000023 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:35.999937 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-34.ec2.internal_2b2a01d40505439c955139cb876044b6/haproxy/0.log" Jul 07 15:06:41.303116 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:41.303074 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-8486594fb6-4nw5x_b2e412e5-2397-4f98-8b6a-b8eb08bb77a4/authorino/0.log" Jul 07 15:06:41.402010 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:41.401982 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-6ngls_56a9a81d-3332-4687-ba1c-c32bb4f8e4bd/kuadrant-console-plugin/0.log" Jul 07 15:06:43.481674 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:43.481591 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c2dff414-68a7-4ae9-8599-3e50e17acfbe/alertmanager/0.log" Jul 07 15:06:43.509663 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:43.509638 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c2dff414-68a7-4ae9-8599-3e50e17acfbe/config-reloader/0.log" Jul 07 15:06:43.536253 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:43.536230 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c2dff414-68a7-4ae9-8599-3e50e17acfbe/kube-rbac-proxy-web/0.log" Jul 07 15:06:43.563139 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:43.563117 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c2dff414-68a7-4ae9-8599-3e50e17acfbe/kube-rbac-proxy/0.log" Jul 07 15:06:43.588727 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:43.588702 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c2dff414-68a7-4ae9-8599-3e50e17acfbe/kube-rbac-proxy-metric/0.log" Jul 07 15:06:43.618482 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:43.618457 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c2dff414-68a7-4ae9-8599-3e50e17acfbe/prom-label-proxy/0.log" Jul 07 15:06:43.646321 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:43.646297 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c2dff414-68a7-4ae9-8599-3e50e17acfbe/init-config-reloader/0.log" Jul 07 15:06:43.728732 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:43.728705 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-s2w66_734e668f-6e19-4fbc-989e-6cdcb9a25532/kube-state-metrics/0.log" Jul 07 15:06:43.748876 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:43.748782 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-s2w66_734e668f-6e19-4fbc-989e-6cdcb9a25532/kube-rbac-proxy-main/0.log" Jul 07 15:06:43.776904 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:43.776876 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-s2w66_734e668f-6e19-4fbc-989e-6cdcb9a25532/kube-rbac-proxy-self/0.log" Jul 07 15:06:43.873568 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:43.873544 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7wxcn_48fa86c9-ae09-45d8-bbdc-558a49fcb7dc/node-exporter/0.log" Jul 07 15:06:43.901310 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:43.901285 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7wxcn_48fa86c9-ae09-45d8-bbdc-558a49fcb7dc/kube-rbac-proxy/0.log" Jul 07 15:06:43.927607 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:43.927582 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7wxcn_48fa86c9-ae09-45d8-bbdc-558a49fcb7dc/init-textfile/0.log" Jul 07 15:06:44.181077 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:44.181049 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-t6gwc_15739c79-527a-466c-98b1-0fa07eb88e62/kube-rbac-proxy-main/0.log" Jul 07 15:06:44.210154 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:44.210128 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-t6gwc_15739c79-527a-466c-98b1-0fa07eb88e62/kube-rbac-proxy-self/0.log" Jul 07 15:06:44.236677 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:44.236645 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-t6gwc_15739c79-527a-466c-98b1-0fa07eb88e62/openshift-state-metrics/0.log" Jul 07 15:06:44.279972 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:44.279943 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89ca039b-62fb-4c67-ab1c-57c07a6838c7/prometheus/0.log" Jul 07 15:06:44.311145 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:44.311119 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89ca039b-62fb-4c67-ab1c-57c07a6838c7/config-reloader/0.log" Jul 07 15:06:44.343708 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:44.343687 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89ca039b-62fb-4c67-ab1c-57c07a6838c7/thanos-sidecar/0.log" Jul 07 15:06:44.390686 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:44.390660 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89ca039b-62fb-4c67-ab1c-57c07a6838c7/kube-rbac-proxy-web/0.log" Jul 07 15:06:44.428394 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:44.428350 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89ca039b-62fb-4c67-ab1c-57c07a6838c7/kube-rbac-proxy/0.log" Jul 07 15:06:44.458824 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:44.458754 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89ca039b-62fb-4c67-ab1c-57c07a6838c7/kube-rbac-proxy-thanos/0.log" Jul 07 15:06:44.487659 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:44.487634 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_89ca039b-62fb-4c67-ab1c-57c07a6838c7/init-config-reloader/0.log" Jul 07 15:06:44.526849 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:44.526821 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-smsz7_cc790513-59cf-4bc9-ab29-63daebcb0ed4/prometheus-operator/0.log" Jul 07 15:06:44.562555 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:44.562531 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-smsz7_cc790513-59cf-4bc9-ab29-63daebcb0ed4/kube-rbac-proxy/0.log" Jul 07 15:06:44.748717 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:44.748632 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848b6bcc56-l8q8m_ace66132-5583-448f-9dea-9d1704bb2b18/thanos-query/0.log" Jul 07 15:06:44.781870 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:44.781832 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848b6bcc56-l8q8m_ace66132-5583-448f-9dea-9d1704bb2b18/kube-rbac-proxy-web/0.log" Jul 07 15:06:44.812767 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:44.812743 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848b6bcc56-l8q8m_ace66132-5583-448f-9dea-9d1704bb2b18/kube-rbac-proxy/0.log" Jul 07 15:06:44.853143 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:44.853117 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848b6bcc56-l8q8m_ace66132-5583-448f-9dea-9d1704bb2b18/prom-label-proxy/0.log" Jul 07 15:06:44.882348 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:44.882327 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848b6bcc56-l8q8m_ace66132-5583-448f-9dea-9d1704bb2b18/kube-rbac-proxy-rules/0.log" Jul 07 15:06:44.912332 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:44.912305 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-848b6bcc56-l8q8m_ace66132-5583-448f-9dea-9d1704bb2b18/kube-rbac-proxy-metrics/0.log" Jul 07 15:06:45.726640 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.726606 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh"] Jul 07 15:06:45.727123 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.727042 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="486a1e57-5fd6-446b-b0f8-85fa847b7489" containerName="maas-api" Jul 07 15:06:45.727123 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.727062 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="486a1e57-5fd6-446b-b0f8-85fa847b7489" containerName="maas-api" Jul 07 15:06:45.727123 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.727076 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5ef9286-65f6-4342-87cc-aa29c0c01eac" containerName="main" Jul 07 15:06:45.727123 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.727084 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5ef9286-65f6-4342-87cc-aa29c0c01eac" containerName="main" Jul 07 15:06:45.727123 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.727114 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5393d83-054f-4195-a8d8-c6b0b2acb70d" containerName="main" Jul 07 15:06:45.727123 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.727122 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5393d83-054f-4195-a8d8-c6b0b2acb70d" containerName="main" Jul 07 15:06:45.727462 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.727219 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="486a1e57-5fd6-446b-b0f8-85fa847b7489" containerName="maas-api" Jul 07 15:06:45.727462 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.727235 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5393d83-054f-4195-a8d8-c6b0b2acb70d" containerName="main" Jul 07 15:06:45.727462 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.727249 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5ef9286-65f6-4342-87cc-aa29c0c01eac" containerName="main" Jul 07 15:06:45.730559 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.730540 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" Jul 07 15:06:45.733082 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.733063 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qxj2b\"/\"kube-root-ca.crt\"" Jul 07 15:06:45.733831 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.733817 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qxj2b\"/\"openshift-service-ca.crt\"" Jul 07 15:06:45.733917 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.733832 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qxj2b\"/\"default-dockercfg-q76xq\"" Jul 07 15:06:45.744510 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.744488 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh"] Jul 07 15:06:45.830233 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.830199 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f99b720b-9ca3-407e-819d-0319a1f46851-podres\") pod \"perf-node-gather-daemonset-9t8dh\" (UID: \"f99b720b-9ca3-407e-819d-0319a1f46851\") " pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" Jul 07 15:06:45.830233 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.830238 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f99b720b-9ca3-407e-819d-0319a1f46851-sys\") pod \"perf-node-gather-daemonset-9t8dh\" (UID: \"f99b720b-9ca3-407e-819d-0319a1f46851\") " pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" Jul 07 15:06:45.830473 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.830313 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnhsv\" (UniqueName: \"kubernetes.io/projected/f99b720b-9ca3-407e-819d-0319a1f46851-kube-api-access-vnhsv\") pod \"perf-node-gather-daemonset-9t8dh\" (UID: \"f99b720b-9ca3-407e-819d-0319a1f46851\") " pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" Jul 07 15:06:45.830473 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.830362 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f99b720b-9ca3-407e-819d-0319a1f46851-proc\") pod \"perf-node-gather-daemonset-9t8dh\" (UID: \"f99b720b-9ca3-407e-819d-0319a1f46851\") " pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" Jul 07 15:06:45.830473 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.830413 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f99b720b-9ca3-407e-819d-0319a1f46851-lib-modules\") pod \"perf-node-gather-daemonset-9t8dh\" (UID: \"f99b720b-9ca3-407e-819d-0319a1f46851\") " pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" Jul 07 15:06:45.931762 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.931732 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f99b720b-9ca3-407e-819d-0319a1f46851-podres\") pod \"perf-node-gather-daemonset-9t8dh\" (UID: \"f99b720b-9ca3-407e-819d-0319a1f46851\") " pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" Jul 07 15:06:45.931933 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.931772 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f99b720b-9ca3-407e-819d-0319a1f46851-sys\") pod \"perf-node-gather-daemonset-9t8dh\" (UID: \"f99b720b-9ca3-407e-819d-0319a1f46851\") " pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" Jul 07 15:06:45.931933 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.931832 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnhsv\" (UniqueName: \"kubernetes.io/projected/f99b720b-9ca3-407e-819d-0319a1f46851-kube-api-access-vnhsv\") pod \"perf-node-gather-daemonset-9t8dh\" (UID: \"f99b720b-9ca3-407e-819d-0319a1f46851\") " pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" Jul 07 15:06:45.931933 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.931886 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f99b720b-9ca3-407e-819d-0319a1f46851-proc\") pod \"perf-node-gather-daemonset-9t8dh\" (UID: \"f99b720b-9ca3-407e-819d-0319a1f46851\") " pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" Jul 07 15:06:45.931933 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.931910 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f99b720b-9ca3-407e-819d-0319a1f46851-lib-modules\") pod \"perf-node-gather-daemonset-9t8dh\" (UID: \"f99b720b-9ca3-407e-819d-0319a1f46851\") " pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" Jul 07 15:06:45.931933 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.931914 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f99b720b-9ca3-407e-819d-0319a1f46851-sys\") pod \"perf-node-gather-daemonset-9t8dh\" (UID: \"f99b720b-9ca3-407e-819d-0319a1f46851\") " pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" Jul 07 15:06:45.931933 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.931910 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f99b720b-9ca3-407e-819d-0319a1f46851-podres\") pod \"perf-node-gather-daemonset-9t8dh\" (UID: \"f99b720b-9ca3-407e-819d-0319a1f46851\") " pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" Jul 07 15:06:45.932180 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.932015 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f99b720b-9ca3-407e-819d-0319a1f46851-proc\") pod \"perf-node-gather-daemonset-9t8dh\" (UID: \"f99b720b-9ca3-407e-819d-0319a1f46851\") " pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" Jul 07 15:06:45.932180 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.932108 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f99b720b-9ca3-407e-819d-0319a1f46851-lib-modules\") pod \"perf-node-gather-daemonset-9t8dh\" (UID: \"f99b720b-9ca3-407e-819d-0319a1f46851\") " pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" Jul 07 15:06:45.941220 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:45.941197 2562 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnhsv\" (UniqueName: \"kubernetes.io/projected/f99b720b-9ca3-407e-819d-0319a1f46851-kube-api-access-vnhsv\") pod \"perf-node-gather-daemonset-9t8dh\" (UID: \"f99b720b-9ca3-407e-819d-0319a1f46851\") " pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" Jul 07 15:06:46.040938 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:46.040853 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" Jul 07 15:06:46.173461 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:46.173435 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh"] Jul 07 15:06:46.175660 ip-10-0-128-34 kubenswrapper[2562]: W0707 15:06:46.175631 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf99b720b_9ca3_407e_819d_0319a1f46851.slice/crio-d22f65f808e42788578fb347568cb0e5187b1371a9d21f6afd5f59942533928f WatchSource:0}: Error finding container d22f65f808e42788578fb347568cb0e5187b1371a9d21f6afd5f59942533928f: Status 404 returned error can't find the container with id d22f65f808e42788578fb347568cb0e5187b1371a9d21f6afd5f59942533928f Jul 07 15:06:46.884319 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:46.884286 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" event={"ID":"f99b720b-9ca3-407e-819d-0319a1f46851","Type":"ContainerStarted","Data":"616a26fd2bf803830134cd1a81b393569b4c92296ee3aed9df5e7b96cb3807a1"} Jul 07 15:06:46.884728 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:46.884335 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" event={"ID":"f99b720b-9ca3-407e-819d-0319a1f46851","Type":"ContainerStarted","Data":"d22f65f808e42788578fb347568cb0e5187b1371a9d21f6afd5f59942533928f"} Jul 07 15:06:46.884728 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:46.884424 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" Jul 07 15:06:46.905354 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:46.905304 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" podStartSLOduration=1.905288724 podStartE2EDuration="1.905288724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 15:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 15:06:46.903404026 +0000 UTC m=+2530.712951516" watchObservedRunningTime="2026-07-07 15:06:46.905288724 +0000 UTC m=+2530.714836214" Jul 07 15:06:49.362522 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:49.362496 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d45ft_e9b2e306-1295-4e9f-8455-dbaa646ccae0/dns/0.log" Jul 07 15:06:49.397642 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:49.397615 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d45ft_e9b2e306-1295-4e9f-8455-dbaa646ccae0/kube-rbac-proxy/0.log" Jul 07 15:06:49.642357 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:49.642251 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8xs5n_64b3ea37-09bc-491f-8e4c-de9cc97e7b5a/dns-node-resolver/0.log" Jul 07 15:06:50.433931 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:50.433903 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zc4j9_b22731b9-be09-4ad5-96ae-dce04203e0ad/node-ca/0.log" Jul 07 15:06:51.859045 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:51.859015 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f67c6995-xtj92_7a2df886-cb1c-43fa-87ef-067cc88830b7/kube-auth-proxy/0.log" Jul 07 15:06:52.004013 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:52.003982 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-5vgm2_f202d8f1-a537-4765-9352-8a229fb5e65f/payload-pre-processing/0.log" Jul 07 15:06:52.899230 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:52.899200 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qxj2b/perf-node-gather-daemonset-9t8dh" Jul 07 15:06:53.000565 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:53.000529 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-btsnj_da4cacc9-a04a-49e6-aef4-7417f001222c/serve-healthcheck-canary/0.log" Jul 07 15:06:53.926272 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:53.926245 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mhwfk_d642794f-888e-40d7-9d1c-ded85fa8153f/kube-rbac-proxy/0.log" Jul 07 15:06:53.978464 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:53.978405 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mhwfk_d642794f-888e-40d7-9d1c-ded85fa8153f/exporter/0.log" Jul 07 15:06:54.014691 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:54.014668 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mhwfk_d642794f-888e-40d7-9d1c-ded85fa8153f/extractor/0.log" Jul 07 15:06:56.229293 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:56.229256 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f67c6995-xtj92_7a2df886-cb1c-43fa-87ef-067cc88830b7/kube-auth-proxy/0.log" Jul 07 15:06:56.312362 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:56.312329 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-5vgm2_f202d8f1-a537-4765-9352-8a229fb5e65f/payload-pre-processing/0.log" Jul 07 15:06:56.950283 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:56.950263 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-64f64495c-wq6t7_3a127701-b325-405f-a500-348b93eefae7/maas-api/0.log" Jul 07 15:06:57.584154 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:57.584065 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f546bc4-m4psb_64f0cd1e-7248-4696-8782-1b544a20848b/manager/0.log" Jul 07 15:06:57.626093 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:57.626067 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6f546bc4-m4psb_64f0cd1e-7248-4696-8782-1b544a20848b/copy-manifests/0.log" Jul 07 15:06:57.738942 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:57.738916 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f67c6995-xtj92_7a2df886-cb1c-43fa-87ef-067cc88830b7/kube-auth-proxy/0.log" Jul 07 15:06:57.770745 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:57.770704 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-cvhmk_ecf59abb-a58d-439b-901c-7beab9368363/postgres/0.log" Jul 07 15:06:57.828348 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:57.828323 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-5vgm2_f202d8f1-a537-4765-9352-8a229fb5e65f/payload-pre-processing/0.log" Jul 07 15:06:59.141188 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:59.141156 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5b9884d45d-lxtjt_b5ab5b50-afc3-4698-8b88-7a548a0ff08c/manager/0.log" Jul 07 15:06:59.176852 ip-10-0-128-34 kubenswrapper[2562]: I0707 15:06:59.176822 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-lxfgc_33989fec-98b5-4ae9-b339-9e835c24d617/openshift-lws-operator/0.log"